Can Lottery Incentives Boost Web Survey Response Rates? Findings from Four Experiments

Institutions of higher education rely on student surveys for a number of purposes, including planning, assessment, and research. Web surveys are especially prevalent given their ease of use and low-cost; yet, obtaining a high response rate is a challenge. Although researchers have investigated the use of incentives in traditional mail surveys, studies investigating the impact of lottery incentives on web survey response rates are limited. In this study, four separate web survey experiments were conducted to measure the effectiveness of lottery-based incentives in a college student population in the United States. Findings reveal that the lottery incentives not only positively impacted response rates but also exerted differential effects by gender. The results have practical implications for higher education researchers who conduct web surveys of college students.     

The Impact of Lottery Incentives on Student Survey Response Rates

Lottery incentives are widely used by institutional researchers despite a lack of research documenting the effectiveness of postpaid incentives in general and lottery incentives in particular. A controlled experiment tested the effects of lottery incentives using a prospective college applicant Web survey, with e-mails sent to more than 9,000 high school students. The impact of the level of lottery incentive on response rates and response bias is discussed.     

Improving alumni survey response rates: An experiment and cost-benefit analysis

Mail surveys are frequently used in higher education research as a means of collecting data relevant for college decision makers. Despite their prevalence, mail surveys have drawbacks, chief among them the potential for low response rates, which may compromise the credibility of research results and diminish their usefulness. Therefore, it is important for institutional researchers to plan and conduct mail surveys that achieve optimal response rates, especially in populations (i.e., alumni) where low response rates may be a problem. This research tested the effect of the survey procedures suggested by Dillman's (1978) Total Design Method on response rate to a mail survey of two-year college alumni. The method used was an experiment with four groups that varied in their degree of adherence to Dillman's procedures, i.e., amount of follow-up and degree of personalized approach. Subjects were randomly assigned to groups. Results provided a test of Dillman's techniques in an educational setting, further information for institutional researchers about ways to improve response rates, and an analysis of the costs and benefits of using Dillman's methods.     

A methodological review of statistical methods for handling multilevel non-nested longitudinal data in educational research

As applications of multilevel modelling in educational research increase, researchers realize that multilevel data collected in many educational settings are often not purely nested. The most common multilevel non-nested data structure is one that involves student mobility in longitudinal studies. This article provides a methodological review of three statistical methods for handling student mobility in longitudinal studies: a multilevel approach, a cross-classified approach, and a cross-classified multiple membership approach. The strengths and weaknesses of each approach and the essential differences between the three approaches are discussed. The Early Childhood Longitudinal Study Kindergarten Cohort data are analysed to demonstrate the differences in parameter estimates and statistical inference between the three approaches. Potential applications of the three approaches in educational research and beyond and directions for further methodological investigations are discussed.     

Exploring ethical issues associated with using online surveys in educational research

Online surveys are increasingly used in educational research, yet little attention has focused on ethical issues associated with their use in educational settings. Here, we draw on the broader literature to discuss 5 key ethical issues in the context of educational survey research: dual teacher/researcher roles; informed consent; use of incentives; privacy, anonymity, and confidentiality; and data quality. We illustrate methods of addressing these issues with our experiences conducing online surveys in educational contexts. Moving beyond the procedural ethics approach commonly adopted in quantitative educational research, we recommend adopting a situated/process ethics approach to identify and respond to ethical issues that may arise during the conduct, analysis, and reporting of online survey research. The benefits of online surveying in comparison to traditional survey methods are highlighted, including the potential for online surveys to provide ethically defensible methods of conducting research that would not be feasible in offline education research settings.     

Meaningful integration within mixed methods studies: Identifying why,what, when,and how

This special issue of Contemporary Educational Psychology brings together an important collection of examples of how researchers have applied mixed methods research to study questions of interest in the field of educational psychology. Collectively, the articles illustrate the unique insights that can be gained when researchers intentionally integrate quantitative and qualitative methods within mixed methods studies. Further, the special issue highlights specific mixed methods strategies that the authors used to achieve and communicate the integration aspects within their studies. The authors’ application of these strategies provides useful models for scholars planning mixed methods studies and scholars reviewing reports of mixed methods studies within educational psychology and beyond.     

Multiple methods: research methods in education projects at NSF1

Projects on science and mathematics education research supported by the National Science Foundation (US government) rarely employ a single method of study. Studies of educational practices that use experimental design are very rare. The most common research method is the case study method and the second most common is some form of experimental design. However, most studies use a combination of methods to study educational practices such as case study and survey. The types of methods found in this group of education researchers represent the best quality available to the education community. The proposals were selected from those carefully reviewed by a panel of senior researchers.     

教育现象学的提问方式与致思路径

问题的提出需要有一种知识论基础作为内在结构支撑.不同研究范式规定了不同的致思理路,形成了对同一教育现象的多元研究视角.教育现象学提问方式从教育事实出发,为研究者提供了一种灵活地"看"教育和"问"教育的方式.教育现象学的问题意识引导研究者对教育现象的意义追问,教育现象学问题域则规定了研究范围和研究范式.教育现象学透过对问题的表述、分析与理解,实现教育意义的重构.以实践取向为基础的教育现象学,通过对教育现象"是其所是"的追问,形成一条深入"教育实事本身"的致思路径.     

Gathering faculty teaching evaluations by in‐class and online surveys: their effects on response rates and evaluations

This study compares student evaluations of faculty teaching that were completed in‐class with those collected online. The two methods of evaluation were compared on response rates and on evaluation scores. In addition, this study investigates whether treatments or incentives can affect the response to online evaluations. It was found that the response rate to the online survey was generally lower than that to the in‐class survey. When a grade incentive was used to encourage response to the online survey, a response rate was achieved that was comparable with that to the in‐class survey. Additionally, the study found that online evaluations do not produce significantly different mean evaluation scores than traditional in‐class evaluations, even when different incentives are offered to students who are asked to complete online evaluations.     

Comparative education research framed by neo-institutional theory: a review of diverse approaches and conflicting assumptions

The rise in globalisation studies in comparative education places neo-institutional theory at the centre of many debates among comparative education researchers. However, uncertainty about how to interpret neo-institutional theory still persists among educational comparativists. With this uncertainty comes misinterpretation of its principles, variations and explanatory power. Two problematic misconceptions prevail: (1) the belief that the ‘world culture’ strand is the only version of neo-institutional theory applicable to comparative education research; and (2) the assumption that the global homogenisation of society, culture and schooling is a goal of researchers applying neo-institutional theory to comparative education phenomena. This article addresses these misconceptions, elucidating neo-institutional theory and its applicability to comparative education research. Our findings suggest that neo-institutional frameworks for comparative education research are useful, but that complementary approaches and methods are also necessary.     

BOOK REVIEW

Critical reviews from “outside”, notably educational sociologists arguing mainly from a British context, have caused some ripples, and maybe even waves, among school effectiveness researchers. To a large extent these external criticisms and the overall nature of the response from school effectiveness researchers are neatly summarized in the following quote from Townsend: “be like us”, say the critics, and the answer is “no thanks”. In this article the arguments form the “external” critics and the response from school effectiveness researchers will not be repeated. Some of the topics in the debate will be revisited, however. The first one is the discussion with respect to the impact of “contextual” or composition effects concerning the average socioeconomic background of students in schools and classrooms. This is one area, that bears upon the foundations of the school effectiveness concept, although it is not, as the critics would have it, a neglected area. The second one concerns conceptualization and theoretical explanation of school effectiveness, as the debate may not have been sufficiently explicit on this issue. The rest of the paper deals with “foundational” issues in school effectiveness research that have not been settled decisively and with changes in perspectives on learning and instruction and educational technology that provide serious challenges. On these issues there is reason for self-criticism and realism in the way school effectiveness research can make progress.     

The Pursuit of School Effectiveness and Educational Quality in Hong Kong

This article briefly describes the Hong Kong education system, discusses the policies and efforts for education quality and school improvement, and reviews the development of school effectiveness research in Hong Kong. Hong Kong people have shown their strong commitment to educational reform for better societal development in the coming decades. A number of important policies have been issued to improve educational practice and school management. Even though some drawbacks and difficulties have inevitably been encountered in policy formulation and implementation, numerous good opportunities have been created for policy-makers, school practitioners, and educational researchers to conduct educational innovation, school improvement, and school effectiveness research. The ongoing education experiments, reform experiences, improvement practices, and effectiveness studies at both the system and school levels may not only benefit Hong Kong people but also make a contribution to international concern for school effectiveness and improvement.     

Propensity Score Matching for Education Data: Worked Examples

Abstract

Randomized controlled trials are not always feasible in educational research, so researchers must use alternative methods to study treatment effects. Propensity score matching is one such method for observational studies that has shown considerable growth in popularity since it was first introduced in the early 1980s. This paper outlines the concept of propensity scores by explaining their theoretical principles and providing two examples of their usefulness within the realm of educational research. Through worked examples, we highlight the effectiveness of propensity scores as a method for reducing bias and increasing the balance between treatment and comparison groups. To aid in the understanding and future use of propensity scores, we provide R syntax for all our analyses.     

教育研究中扎根理论的价值本真与方法祛魅

扎根理论作为一种质性研究范式,自21世纪初被引至国内教育研究领域后,便得到广泛应用。就实践论、方法论、本体论三重逻辑而言,教育研究中扎根理论应然的价值表征为以价值与事实的勾连,化解教育研究危机;以定量方法和定性方法的调和,缓冲教育研究的方法范式之争;以艺术与科学的融合,廓清教育学的本质属性。然而,在相关研究的实然样态下,扎根理论在实践论、方法论、本体论层面上分别暴露出应用形式桎梏、人文情怀缺位及扎根精神淡薄的问题。为此,教育研究者亟需对扎根理论进行祛魅,一要创生扎根理论的"时代风格",助推实践过程的"理论自觉";二要重塑扎根理论的"人文情怀",警惕方法层面的"花拳绣腿";三要重视研究过程的"扎根精神",回归扎根理论的"初心终旨",以实现扎根理论在我国教育研究中应然价值本真与实然价值追求的统一。     

Second Life的教育应用现状及其功能述评

Second Life虚拟世界自2003年问世,获得了语言学、商业、医学、教育学等众多领域研究者青睐,成为其理论与实践研究的创新视角。当前国内外SL教育应用现状及其功能的研究成果主要聚焦于5个方面:(1)SL可作为多学科教育教学的实践环境;(2)SL提供不同学习情境体验与能力迁移;(3)SL可作为网络学习社区;(4)SL提供多样化教学实践形式;(5)SL提供创新性教学活动设计。当然SL教育应用研究也存在一定缺陷,如技术门槛、硬件需求、活动设计以及资源创设等方面。未来的SL教育应用研究将会在研究方法、呈现工具、技术强度以及沉浸体验等方面获得更多的关注。     

Effect sizes as result interpretation aids in single‐subject experimental research: description and application of four nonoverlap methods

Single‐subject experimental research (SSER), one of the most commonly used research methods in special education and applied behaviour analysis, is a scientific, rigorous and valid method to evaluate the effectiveness of behavioural, educational and psychological treatments. However, studies using single‐subject experimental research designs are often excluded from meta‐analyses of evidence‐based practices due to the lack of methodological consensus on the type of effect size indices to be used to determine treatment effect. To promote the use of effect size indices as an adjunct to visual analysis, this article describes four nonoverlap methods (PND, IRD, PEM‐T and Tau‐U) and demonstrates their application to data obtained from studies employing different SSER designs. Advantages and disadvantages of each method are highlighted and considerations for selecting the most appropriate method are provided for researchers and practitioners.