Assessing Response Rates and Nonresponse Bias in Web and Paper Surveys

Using data collected as part of the second pilot administration of Your First College Year (YFCY), a national survey of first-year college students, this study was designed to examine both response rates and nonresponse bias across four survey administration groups: paper-only, paper with web option, web-only with response incentive, and web-only without response incentive. Findings indicate that response rates vary by mode of administration. Moreover, predictors of response differed by administration group. Results are discussed in light of the recent surge of interest in online survey research.     

The Impact of Lottery Incentives on Student Survey Response Rates

Lottery incentives are widely used by institutional researchers despite a lack of research documenting the effectiveness of postpaid incentives in general and lottery incentives in particular. A controlled experiment tested the effects of lottery incentives using a prospective college applicant Web survey, with e-mails sent to more than 9,000 high school students. The impact of the level of lottery incentive on response rates and response bias is discussed.     

Student Survey Response Rates across Institutions: Why Do they Vary?

While many studies have examined nonresponse in student surveys, little research investigates why some schools achieve higher student survey response rates than other schools. Using hierarchical linear modeling, we analyze survey data from 321 institutions that participated in the 2003 National Survey of Student Engagement to understand how characteristics of colleges and universities relate to student survey response rates. We find that the makeup of the student body, as well as institutional characteristics such public/private status and urban location affects response rates, and that the number of computers per undergraduate has a strong positive effect for web survey response rates.     

A Comparison of Response Rate,Response Time,and Costs of Mail and Electronic Surveys

Response rates, response time, and costs of mail and electronic surveys were compared. The survey sample consisted of college faculty in the southeastern United States. Mail surveys yielded a higher response rate and a lower rate of unde-liverable surveys. However, receipt of responses was significantly quicker for those delivered electronically, with over 80% of initial responses arriving before receipt of the 1st returned mail survey. The costs associated with distribution of surveys and preparation of data for analysis were much lower with the electronically delivered surveys.     

Identifying Disengaged Survey Responses: New Evidence Using Response Time Metadata

Disengaged responding is a phenomenon that often biases observed scores from achievement tests and surveys in practically and statistically significant ways. This problem has led to the development of methods to detect and correct for disengaged responses on both achievement test and survey scores. One major disadvantage when trying to detect disengaged responses on surveys is that, unlike on achievement tests, there are no correct answers. As a result, validating decision rules for detection methods is problematic. In this study, we condition results from a variety of detection methods used to identify disengaged survey responses on response times. We then show how this conditional approach may be useful in identifying where to set response time thresholds for survey items, as well as in avoiding misclassification when using other detection methods.     

Prenotification,Incentives, and Survey Modality: An Experimental Test of Methods to Increase Survey Response Rates of School Principals

Abstract

Teacher and principal surveys are among the most common data collection techniques employed in education research. Yet there is remarkably little research on survey methods in education, or about the most cost-effective way to raise response rates among teachers and principals. In an effort to explore various methods for increasing survey response rates, we randomly assigned 1,177 high school principals in the state of Michigan to 1 of 4 experimental conditions. We varied the mode of survey delivery, the mode in which the prenotification letter was sent, and whether or not a $10 incentive was provided. The results indicate that providing a monetary incentive substantially increased response rates over the no incentive condition and that principals were more likely to respond to a paper-based survey than a web-based one.     

组织沟通对工作压力的影响机制分析——应对方式的中介作用

抽取有代表性的企事业、机关单位在职人员为研究对象,探讨分析了组织沟通各要素与成员工作压力间的关系,并从中介作用的角度,检验并深入分析了员工的应对方式在组织沟通与工作压力效能关系中的中介作用机制,得到的主要结果为:员工积极应对方式在组织沟通与工作行为之间起到部分中介作用.即组织沟通变量一方面直接影响工作行为,一方面又通过个体的应对方式来影响工作行为;另外,积极应对方式在组织沟通与积极心理感受之间起到完全中介作用,即组织沟通变量完全通过积极应对方式来影响积极心理感受.     

Longitudinal Models for Ordinal Data With Many Zeros and Varying Numbers of Response Categories

Ordinal response scales are often used to survey behaviors, including data collected in longitudinal studies. Advanced analytic methods are now widely available for longitudinal data. This study evaluates the performance of 4 methods as applied to ordinal measures that differ by the number of response categories and that include many zeros. The methods considered are hierarchical linear models (HLMs), growth mixture mixed models (GMMMs), latent class growth analysis (LCGA), and 2-part latent growth models (2PLGMs). The methods are evaluated by applying each to empirical response data in which the number of response categories is varied. The methods are applied to each outcome variable, first treating the outcome as continuous and then as ordinal, to compare the performance of the methods given both a different number of response categories and treatment of the variables as continuous versus ordinal. We conclude that although the 2PLGM might be preferred, no method might be ideal.     

Response bias in college freshmen's responses to mail surveys

This study compared the attitudes and demographic characteristics of an intact group of college freshmen to those of a sample responding to a mail survey and a sample responding to a telephone follow-up. Contrary to expectation, the three samples did not differ significantly in mean attitude responses. The absence of significant mean differences indicated no significant bias in college freshmen's responses to a mail survey. However, a higher percentage of returns was received from females and higher achievers than from males and low achievers.     

Validation of Cognitive Sensitivity for Item Response Curves

This study sought a scientific way to examine whether item response curves are influenced systematically by the cognitive processes underlying solution of the items in a procedural domain (addition of fractions). Starting from an expert teacher's logical task analysis and prediction of various erroneous rules and sources of misconceptions, an error diagnostic program was developed. This program was used to carry out an error analysis of test performance by three samples of students. After the cognitive structure of the subtasks was validated by a majority of the students, the items were characterized by their underlying subtask patterns. It was found that item response curves for items in the same categories were significantly more homogeneous than those in different categories. In other words, underlying cognitive subtasks appeared to systematically influence the slopes and difficulties of item response curves.     

新型冠状病毒肺炎疫情时期对高校网络授课的学生评价调查及教学改革的启发

以南宁师范大学物理与电子学院本科生为样本,收集学生对疫情下网络授课的评价,并基于取得的结果对线上教学的教学改革提出建议。本调查对高校网络教学形式的研究、持续抗疫等突发事件的应对提供了参考。     

Can we augment web responses with telephonic responses to a graduate destination survey?

The 2010 Western Cape graduate destination survey utilised a sequential mixed-mode design in which an initial web survey was augmented with an equivalent telephonic survey. This article examines mode effect in the Western Cape survey in terms of overall effect size and the bearing it had on the main outcome of the study. Standardised residuals and Cramér’s V are used to determine mode effect across two scenarios, a full sample vs. a subsample, and using two categorical questions with different numbers of response categories. Overall effect size appears to be small in the first question, but increases noticeably together with non-responses in the second question that has many more response categories. Web responses to alumni or graduate destination surveys can perhaps be augmented with telephonic responses if necessary, provided response categories are kept to a minimum, and interviewers are trained properly and monitored for possible interviewer misbehaviour. The benefit of obtaining larger samples should then also outweigh the benefit of using web surveys alone.     

Response bias in follow-up studies of college students

The problems of response bias in longitudinal studies of college students are examined. An extensive follow-up questionnaire was sent to 1,253 college seniors who had participated in a similar survey as freshman four years earlier. Careful measure of student responsiveness in relation to various techniques designed to increase the proportion of responders (e.g., postcard, telephone contact) were kept.The less responsive groups were significantly different from their more responsive counterparts on nearly a dozen variables representing a wide variety of content areas, including academic achievement, self-concept, alcohol consumption, social deviance, and major choice preferences. Controlling for sex and socioeconomic status served to reduce, but not eliminate, these biases. Overall, the results indicate that researchers cannot account for follow-up nonresponse bias by making statistical adjustments according to data available at initial testing. The results are discussed in light of identifying the reasons for nonresponse, and attempting to develop categories of nonresponders who may be motivated to cooperate by different types of follow-up techniques.     

Explaining Variability in Response Style Traits: A Covariate-Adjusted IRTree

Contamination of responses due to extreme and midpoint response style can confound the interpretation of scores, threatening the validity of inferences made from survey responses. This study incorporated person-level covariates in the multidimensional item response tree model to explain heterogeneity in response style. We include an empirical example and two simulation studies to support the use and interpretation of the model: parameter recovery using Markov chain Monte Carlo (MCMC) estimation and performance of the model under conditions with and without response styles present. Item intercepts mean bias and root mean square error were small at all sample sizes. Item discrimination mean bias and root mean square error were also small but tended to be smaller when covariates were unrelated to, or had a weak relationship with, the latent traits. Item and regression parameters are estimated with sufficient accuracy when sample sizes are greater than approximately 1,000 and MCMC estimation with the Gibbs sampler is used. The empirical example uses the National Longitudinal Study of Adolescent to Adult Health’s sexual knowledge scale. Meaningful predictors associated with high levels of extreme response latent trait included being non-White, being male, and having high levels of parental support and relationships. Meaningful predictors associated with high levels of the midpoint response latent trait included having low levels of parental support and relationships. Item-level covariates indicate the response style pseudo-items were less easy to endorse for self-oriented items, whereas the trait of interest pseudo-items were easier to endorse for self-oriented items.     

Encouraging College Student Active Engagement in Learning: The Influence of Response Methods

The purpose of this study was to examine the use of two student response methods within selected college lecture halls. Kinesiology majors from three universities were asked to respond to questions during two consecutive lectures, one using “clickers” and the other using hand-raising. Participation and comprehension rates were statistically significantly different following the use of the different response methods. Participant survey responses revealed insight into student participation, question presentation, perceived cognitive engagement, and overall learning in traditional lecture settings using the response methods.     

Self-Reported Learning Gains: A Theory and Test of College Student Survey Response

Recent studies have asserted that self-reported learning gains (SRLG) are valid measures of learning, because gains in specific content areas vary across academic disciplines as theoretically predicted. In contrast, other studies find no relationship between actual and self-reported gains in learning, calling into question the validity of SRLG. I reconcile these two divergent sets of literature by proposing a theory of college student survey response that relies on the belief-sampling model of attitude formation. This theoretical approach demonstrates how students can easily construct answers to SRLG questions that will result in theoretically consistent differences in gains across academic majors, while at the same time lacking the cognitive ability to accurately report their actual learning gains. Four predictions from the theory are tested, using data from the 2006–2009 Wabash National Study. Contrary to previous research, I find little evidence as to the construct and criterion validity of SRLG questions.     

Validating a teaching survey which drives increased response rates in a unit survey

At Curtin University, student perceptions of teaching and learning are gathered and reported online through eVALUate, a system that includes separate unit and teaching surveys. This article reports the development and validation of one of those surveys, the eVALUate teaching survey, which was developed based on the research literature on excellent teaching and evaluation. Since its development in 2006, repeated statistical testing using progressively larger samples has shown that the survey is valid and reliable. Moreover, the way in which the teaching survey is deployed within eVALUate, appended to the unit survey, which provides crucial institutional data, has significantly increased university response rates. This validated instrument is used for self-reflection, professional development, and rewarding staff.     

Modeling Response Styles in Cross‐Country Self‐Reports: An Application of a Multilevel Multidimensional Nominal Response Model

We examined the feasibility and results of a multilevel multidimensional nominal response model (ML‐MNRM) for measuring both substantive constructs and extreme response style (ERS) across countries. The ML‐MNRM considers within‐country clustering while allowing overall item slopes to vary across items and examination of whether certain items were more prone to ERS. We applied this model to survey items from TALIS 2013. Results indicated that self‐efficacy items were more likely to trigger ERS compared to need for professional development, and the between‐country relationships among constructs can change due to ERS. Simulations assessed the estimation approach and found adequate recovery of model parameters and factor scores. We stress the importance of additional validity studies to improve the cross‐cultural comparability of substantive constructs.     

Asymptotic Standard Errors for Item Response Theory True Score Equating of Polytomous Items

Building on previous works by Lord and Ogasawara for dichotomous items, this article proposes an approach to derive the asymptotic standard errors of item response theory true score equating involving polytomous items, for equivalent and nonequivalent groups of examinees. This analytical approach could be used in place of empirical methods like the bootstrap method, to obtain standard errors of equated scores. Formulas are introduced to obtain the derivatives for computing the asymptotic standard errors. The approach was validated using mean‐mean, mean‐sigma, random‐groups, or concurrent calibration equating of simulated samples, for tests modeled using the generalized partial credit model or the graded response model.     

Differential Response family assessments: Listening to what parents say about service helpfulness

An increasing number of Child Protective Services (CPS) systems are implementing Differential Response (DR) approaches in which lower-risk families are served through a family assessment response that emphasizes a family centered approach and the provision of concrete and preventative services. Quantitative survey data collected from parents suggests that those who receive family assessments are more engaged, receive more concrete services, and have higher overall satisfaction than those who receive a traditional investigation; yet little is known about which services provided through a family assessment are most helpful to parents. This qualitative study sought input from 20 parents who received a DR family assessment response in order to provide an in-depth analysis of which aspects of their CPS experience they perceived as most helpful. Results suggest that a positive and emotionally supportive relationship with the caseworker was of utmost significance. Other caseworker-provided services were described as helpful, particularly those that helped parents establish or improve relationships with others, including advocacy with other service providers, mediation of family disagreements, and coaching on parenting or relationship skills. Material support, such as providing cash assistance for rent or furniture, was helpful when received but occurred less frequently and was sometimes a source of frustration when accompanied by lengthy waits for assistance. These findings have implications for CPS practice, including enhancing caseworkers’ relationship-building and engagement skills through pre-service educational coursework and in-service trainings; allowing caseworkers adequate time to develop supportive relationships with parents; and reducing the institutional barriers that delay the provision of concrete support.