RESUMEN
Responsive survey design is a technique aimed at improving the efficiency or quality of surveys by using incoming data from the field to make design changes. The technique was pioneered on large national surveys, but the tools can also be applied on the smaller-scale surveys most commonly used by sociologists. We demonstrate responsive survey design in a small-scale, list-based sample survey of students on the topic of sexual misconduct. We investigate the impact of individual incentive levels and a two-phase responsive design with changes to mode of contact as approaches for limiting the potential of nonresponse bias in data from such surveys. Our analyses demonstrate that a two-phase design introducing telephone and face-to-face reminders to complete the survey can produce stronger change in response rates and characteristics of those who respond than higher incentive levels. These findings offer tools for sociologists designing smaller-scale surveys of special populations or sensitive topics.
RESUMEN
BACKGROUND: In health research, population estimates are generally obtained from probability-based surveys. In market research surveys are frequently conducted from volunteer web panels. Propensity score adjustment (PSA) is often used at analysis to try to remove bias in the web survey, but empirical evidence of its effectiveness is mixed. We assess the ability of PSA to remove bias in the context of sensitive sexual health research and the potential of web panel surveys to replace or supplement probability surveys. METHODS: Four web panel surveys asked a subset of questions from the third British National Survey of Sexual Attitudes and Lifestyles (Natsal-3). Five propensity scores were generated for each web survey. The scores were developed from progressively larger sets of variables, beginning with demographic variables only and ending with demographic, sexual identity, lifestyle, attitudinal and sexual behaviour variables together. The surveys were weighted to match Natsal-3 based on propensity score quintiles. The performance of each survey and weighting was assessed by calculating the average 'absolute' odds ratio (inverse of the odds ratio if less than 1) across 22 pre-specified sexual behaviour outcomes of interest comparing the weighted web survey with Natsal-3. The average standard error across odds ratios was examined to assess the impact of weighting upon variance. RESULTS: Propensity weighting reduced bias relative to Natsal-3 as more variables were added for males, but had little effect for females, and variance increased for some surveys. Surveys with more biased estimates before propensity weighting showed greater reduction in bias from adjustment. Inconsistencies in performance were evident across surveys and outcomes. For most surveys and outcomes any reduction in bias was only partial and for some outcomes the bias increased. CONCLUSIONS: Even after propensity weighting using a rich range of information, including some sexual behaviour variables, some bias remained and variance increased for some web surveys. Whilst our findings support the use of PSA for web panel surveys, the reduction in bias is likely to be partial and unpredictable, consistent with the findings from market research. Our results do not support the use of volunteer web panels to generate unbiased population health estimates.
Asunto(s)
Conductas Relacionadas con la Salud , Conducta Sexual , Femenino , Encuestas Epidemiológicas , Humanos , Masculino , Puntaje de Propensión , Sesgo de SelecciónRESUMEN
BACKGROUND: The All of Us Research Program is building a national longitudinal cohort and collecting data from multiple information sources (e.g., biospecimens, electronic health records, and mobile/wearable technologies) to advance precision medicine. Participant-provided information, collected via surveys, will complement and augment these information sources. We report the process used to develop and refine the initial three surveys for this program. METHODS: The All of Us survey development process included: (1) prioritization of domains for scientific needs, (2) examination of existing validated instruments, (3) content creation, (4) evaluation and refinement via cognitive interviews and online testing, (5) content review by key stakeholders, and (6) launch in the All of Us electronic participant portal. All content was translated into Spanish. RESULTS: We conducted cognitive interviews in English and Spanish with 169 participants, and 573 individuals completed online testing. Feedback led to over 40 item content changes. Lessons learned included: (1) validated survey instruments performed well in diverse populations reflective of All of Us; (2) parallel evaluation of multiple languages can ensure optimal survey deployment; (3) recruitment challenges in diverse populations required multiple strategies; and (4) key stakeholders improved integration of surveys into larger Program context. CONCLUSIONS: This efficient, iterative process led to successful testing, refinement, and launch of three All of Us surveys. Reuse of All of Us surveys, available at http://researchallofus.org, may facilitate large consortia targeting diverse populations in English and Spanish to capture participant-provided information to supplement other data, such as genetic, physical measurements, or data from electronic health records.
Asunto(s)
Encuestas Epidemiológicas/métodos , Medicina de Precisión , Adolescente , Adulto , Anciano , Anciano de 80 o más Años , Análisis Factorial , Femenino , Humanos , Estudios Longitudinales , Masculino , Persona de Mediana Edad , Proyectos Piloto , Investigación Cualitativa , Traducciones , Estados Unidos , Adulto JovenRESUMEN
Social processes that change quickly are difficult to study, because they require frequent survey measurement. Weekly, daily, or even hourly measurement may be needed depending on the topic. With more frequent measurement comes the prospect of more complex patterns of missing data. The mechanisms creating the missing data may be varied, ranging from technical issues such as lack of an Internet connection to refusal to complete a requested survey. We examine one approach to mitigating the damage of these missing data - a follow-up or closeout interview that is completed after the frequent measurement. The Relationship Dynamics and Social Life (RDSL) study used this approach. The study asked women weekly about their attitudes and behaviors related to sexual relationships and pregnancy. The surveys were carried out for 130 weeks and concluded with a closeout interview. We explore the patterns of missing data in the RDSL study and then examine associations between the data collected in the closeout survey and key variables collected in the weekly survey. We then assess the extent to which data from the closeout survey are useful in repairing the damage caused by missing data.
Asunto(s)
Exactitud de los Datos , Recolección de Datos/normas , Estudios de Seguimiento , Entrevistas como Asunto/normas , Estudios Longitudinales , Proyectos de Investigación/normas , Adulto , Anciano , Anciano de 80 o más Años , Recolección de Datos/estadística & datos numéricos , Femenino , Humanos , Entrevistas como Asunto/estadística & datos numéricos , Masculino , Persona de Mediana Edad , Proyectos de Investigación/estadística & datos numéricosRESUMEN
Challenges to survey data collection have increased the costs of social research via face-to-face surveys so much that it may become extremely difficult for social scientists to continue using these methods. A key drawback to less expensive Internet-based alternatives is the threat of biased results from coverage errors in survey data. The rise of Internet-enabled smartphones presents an opportunity to re-examine the issue of Internet coverage for surveys and its implications for coverage bias. Two questions (on Internet access and smartphone ownership) were added to the National Survey of Family Growth (NSFG), a U.S. national probability survey of women and men age 15-44, using a continuous sample design. We examine 16 quarters (4 years) of data, from September 2012 to August 2016. Overall, we estimate that 82.9% of the target NSFG population has Internet access, and 81.6% has a smartphone. Combined, this means that about 90.7% of U.S. residents age 15-44 have Internet access, via either traditional devices or a smartphone. We find some evidence of compensatory coverage when looking at key race/ethnicity and age subgroups. For instance, while Black teens (15-18) have the lowest estimated rate of Internet access (81.9%) and the lowest rate of smartphone usage (72.6%), an estimated 88.0% of this subgroup has some form of Internet access. We also examine the socio-demographic correlates of Internet and smartphone coverage, separately and combined, as indicators of technology access in this population. In addition, we look at the effect of differential coverage on key estimates produced by the NSFG, related to fertility, family formation, and sexual activity. While this does not address nonresponse or measurement biases that may differ for alternative modes, our paper has implications for possible coverage biases that may arise when switching to a Web-based mode of data collection, either for follow-up surveys or to replace the main face-to-face data collection.
RESUMEN
We study the influence of frequent survey measurement on behavior. Widespread access to the Internet has made important breakthroughs in frequent measurement possible-potentially revolutionizing social science measurement of processes that change quickly over time. One key concern about using such frequent measurement is that it may influence the behavior being studied. We investigate this possibility using both a population-based experiment with random assignment to participation in a weekly journal for twelve months (versus no journal) and a large-scale, population-based, journal-keeping study with weekly measurement for 30 months. Results reveal few of the measured behaviors are correlated with assignment to frequent measurement. Theoretical reasoning regarding the likely behavioral response to frequent measurement correctly predicts domains most vulnerable to this possibility. Overall, however, we found little evidence of behavioral response to frequent measurement.
Asunto(s)
Autoinforme , Conducta Sexual/psicología , Adulto , Recolección de Datos/métodos , Femenino , Humanos , Internet , Adulto JovenRESUMEN
We introduce this special issue on the critical matter of whether the existing household panel surveys in the U.S. are adequate to address the important emerging social science and policy questions of the next few decades. We summarize the conference papers which address this issue in different domains. The papers detail many new and important emerging research questions but also identify key limitations in existing panels in addressing those questions. To address these limitations, we consider the advantages and disadvantages of initiating a new, general-purpose omnibus household panel in the U.S. We also discuss the particular benefits of starting new panels that have specific targeted domains such as child development, population health and health care. We also develop a list of valuable enhancements to existing panels which could address many of their limitations.
RESUMEN
BACKGROUND: Many decisions can be understood in terms of actors' valuations of benefits and costs. The article investigates whether this is also true of patient medical decision making. It aims to investigate (i) the importance patients attach to various reasons for and against nine medical decisions; (ii) how well the importance attached to benefits and costs predicts action or inaction; and (iii) how such valuations are related to decision confidence. METHODS: In a national random digit dial telephone survey of U.S. adults, patients rated the importance of various reasons for and against medical decisions they had made or talked to a health-care provider about during the past 2 years. Participants were 2575 English-speaking adults age 40 and older. Data were analysed by means of logistic regressions predicting action/inaction and linear regressions predicting confidence. RESULTS: Aggregating individual reasons into those that may be regarded as benefits and those that may be regarded as costs, and weighting them by their importance to the patient, shows the expected relationship to action. Perceived benefits and costs are also significantly related to the confidence patients report about their decision. CONCLUSION: The factors patients say are important in their medical decisions reflect a subjective weighing of benefits and costs and predict action/inaction although they do not necessarily indicate that patients are well informed. The greater the difference between the importance attached to benefits and costs, the greater patients' confidence in their decision.
Asunto(s)
Toma de Decisiones , Cooperación del Paciente/psicología , Participación del Paciente , Percepción , Adulto , Análisis Costo-Beneficio , Detección Precoz del Cáncer/psicología , Escolaridad , Femenino , Humanos , Masculino , Persona de Mediana Edad , Prioridad del Paciente , Medicamentos bajo Prescripción/administración & dosificación , Medicamentos bajo Prescripción/efectos adversos , Procedimientos Quirúrgicos Operativos/psicología , Estados UnidosRESUMEN
BACKGROUND: Nonprobability Web surveys using volunteer panels can provide a relatively cheap and quick alternative to traditional health and epidemiological surveys. However, concerns have been raised about their representativeness. OBJECTIVE: The aim was to compare results from different Web panels with a population-based probability sample survey (n=8969 aged 18-44 years) that used computer-assisted self-interview (CASI) for sensitive behaviors, the third British National Survey of Sexual Attitudes and Lifestyles (Natsal-3). METHODS: Natsal-3 questions were included on 4 nonprobability Web panel surveys (n=2000 to 2099), 2 using basic quotas based on age and sex, and 2 using modified quotas based on additional variables related to key estimates. Results for sociodemographic characteristics were compared with external benchmarks and for sexual behaviors and opinions with Natsal-3. Odds ratios (ORs) were used to express differences between the benchmark data and each survey for each variable of interest. A summary measure of survey performance was the average absolute OR across variables. Another summary measure was the number of key estimates for which the survey differed significantly (at the 5% level) from the benchmarks. RESULTS: For sociodemographic variables, the Web surveys were less representative of the general population than Natsal-3. For example, for men, the average absolute OR for Natsal-3 was 1.14, whereas for the Web surveys the average absolute ORs ranged from 1.86 to 2.30. For all Web surveys, approximately two-thirds of the key estimates of sexual behaviors were different from Natsal-3 and the average absolute ORs ranged from 1.32 to 1.98. Differences were appreciable even for questions asked by CASI in Natsal-3. No single Web survey performed consistently better than any other did. Modified quotas slightly improved results for men, but not for women. CONCLUSIONS: Consistent with studies from other countries on less sensitive topics, volunteer Web panels provided appreciably biased estimates. The differences seen with Natsal-3 CASI questions, where mode effects may be similar, suggest a selection bias in the Web surveys. The use of more complex quotas may lead to some improvement, but many estimates are still likely to differ. Volunteer Web panels are not recommended if accurate prevalence estimates for the general population are a key objective.
Asunto(s)
Actitud , Encuestas Epidemiológicas/métodos , Conducta Sexual , Adolescente , Adulto , Femenino , Humanos , Internet , Estilo de Vida , Masculino , Persona de Mediana Edad , Prevalencia , Muestreo , Adulto JovenRESUMEN
The American Family Health Study (AFHS) collected family health and fertility data from a national probability sample of persons aged 18-49 between September 2021 and May 2022, using web and mail exclusively. In July 2022, we surveyed AFHS respondents and gauged their willingness to become part of a national web panel that would create novel longitudinal data on these topics. We focus on predictors of willingness to participate, identifying the potential selection bias that this type of approach may introduce. We found that efforts of this type to create a national web panel may introduce potential selection bias in estimates based on the panel respondents, with individuals having higher socio-economic status being more cooperative. Thus, alternative recruitment strategies and re-weighting of the subsample may be needed to further reduce selection bias. We present methodological implications of our results, limitations of our approach, and suggestions for further research on this topic.
RESUMEN
Grid or matrix questions are associated with a number of problems in Web surveys. In this paper, we present results from two experiments testing the design of grid questions to reduce breakoffs, missing data, and satisficing. The first examines dynamic elements to help guide respondent through the grid, and on splitting a larger grid into component pieces. The second manipulates the visual complexity of the grid and on simplifying the grid. We find that using dynamic feedback to guide respondents through a multi-question grid helps reduce missing data. Splitting the grids into component questions further reduces missing data and motivated underreporting. The visual complexity of the grid appeared to have little effect on performance.
RESUMEN
In push-to-web surveys that use postal mail to contact sampled cases, participation is contingent on the mail being opened and the survey invitations being delivered. The design of the mailings is crucial to the success of the survey. We address the question of how to design invitation mailings that can grab potential respondents' attention and sway them to be interested in the survey in a short window of time. In the household screening stage of a national survey, the American Family Health Study, we experimentally tested three mailing design techniques for recruiting respondents: (1) a visible cash incentive in the initial mailing, (2) a second incentive for initial nonrespondents, and (3) use of Priority Mail in the nonresponse follow-up mailing. We evaluated the three techniques' overall effects on response rates as well as how they differentially attracted respondents with different characteristics. We found that all three techniques were useful in increasing the screening response rates, but there was little evidence that they had differential effects on sample subgroups that could help to reduce nonresponse biases.
RESUMEN
Survey researchers have carefully modified their data collection operations for various reasons, including the rising costs of data collection and the ongoing Coronavirus disease (COVID-19) pandemic, both of which have made in-person interviewing difficult. For large national surveys that require household (HH) screening to determine survey eligibility, cost-efficient screening methods that do not include in-person visits need additional evaluation and testing. A new study, known as the American Family Health Study (AFHS), recently initiated data collection with a national probability sample, using a sequential mixed-mode mail/web protocol for push-to-web US HH screening (targeting persons aged 18-49 years). To better understand optimal approaches for this type of national screening effort, we embedded two randomized experiments in the AFHS data collection. The first tested the use of bilingual respondent materials where mailed invitations to the screener were sent in both English and Spanish to 50 percent of addresses with a high predicted likelihood of having a Spanish speaker and 10 percent of all other addresses. We found that the bilingual approach did not increase the response rate of high-likelihood Spanish-speaking addresses, but consistent with prior work, it increased the proportion of eligible Hispanic respondents identified among completed screeners, especially among addresses predicted to have a high likelihood of having Spanish speakers. The second tested a form of nonresponse follow-up, where a subsample of active sampled HHs that had not yet responded to the screening invitations was sent a priority mailing with a $5 incentive, adding to the $2 incentive provided for all sampled HHs in the initial screening invitation. We found this approach to be quite valuable for increasing the screening survey response rate.
RESUMEN
Although the potential for participant selection bias is readily acknowledged in the momentary data collection literature, very little is known about uptake rates in these studies or about differences in the people that participate versus those who do not. This study analyzed data from an existing Internet panel of older people (age 50 and greater) who were offered participation into a momentary study (n = 3,169), which made it possible to compute uptake and to compare many characteristics of participation status. Momentary studies present participants with brief surveys multiple times a day over several days; these surveys ask about immediate or recent experiences. A 29.1% uptake rate was observed when all respondents were considered, whereas a 39.2% uptake rate was found when individuals who did not have eligible smartphones (necessary for ambulatory data collection) were eliminated from the analyses. Taking into account the participation rate for being in this Internet panel, we estimate uptake rates for the general population to be about 5%. A consistent pattern of differences emerged between those who accepted the invitation to participate versus those who did not (in univariate analyses): participants were more likely to be female, younger, have higher income, have higher levels of education, rate their health as better, be employed, not be retired, not be disabled, have better self-rated computer skills, and to have participated in more prior Internet surveys (all p < .0026). Many variables were not associated with uptake including race, big five personality scores, and subjective well-being. For several of the predictors, the magnitude of the effects on uptake was substantial. These results indicate the possibility that, depending upon the associations being investigated, person selection bias could be present in momentary data collection studies.
Asunto(s)
Evaluación Ecológica Momentánea , Proyectos de Investigación , Humanos , Femenino , Anciano , Persona de Mediana Edad , Masculino , Sesgo de Selección , Encuestas y Cuestionarios , Teléfono InteligenteRESUMEN
OBJECTIVE: The All of Us Research Program collects data from multiple information sources, including health surveys, to build a national longitudinal research repository that researchers can use to advance precision medicine. Missing survey responses pose challenges to study conclusions. We describe missingness in All of Us baseline surveys. STUDY DESIGN AND SETTING: We extracted survey responses between May 31, 2017, to September 30, 2020. Missing percentages for groups historically underrepresented in biomedical research were compared to represented groups. Associations of missing percentages with age, health literacy score, and survey completion date were evaluated. We used negative binomial regression to evaluate participant characteristics on the number of missed questions out of the total eligible questions for each participant. RESULTS: The dataset analyzed contained data for 334,183 participants who submitted at least one baseline survey. Almost all (97.0%) of the participants completed all baseline surveys, and only 541 (0.2%) participants skipped all questions in at least one of the baseline surveys. The median skip rate was 5.0% of the questions, with an interquartile range (IQR) of 2.5% to 7.9%. Historically underrepresented groups were associated with higher missingness (incidence rate ratio (IRR) [95% CI]: 1.26 [1.25, 1.27] for Black/African American compared to White). Missing percentages were similar by survey completion date, participant age, and health literacy score. Skipping specific questions were associated with higher missingness (IRRs [95% CI]: 1.39 [1.38, 1.40] for skipping income, 1.92 [1.89, 1.95] for skipping education, 2.19 [2.09-2.30] for skipping sexual and gender questions). CONCLUSION: Surveys in the All of Us Research Program will form an essential component of the data researchers can use to perform their analyses. Missingness was low in All of Us baseline surveys, but group differences exist. Additional statistical methods and careful analysis of surveys could help mitigate challenges to the validity of conclusions.
Asunto(s)
Salud Poblacional , Humanos , Encuestas y Cuestionarios , Encuestas Epidemiológicas , Conducta SexualRESUMEN
Survey records are increasingly being linked to administrative databases to enhance the survey data and increase research opportunities for data users. A necessary prerequisite to linking survey and administrative records is obtaining informed consent from respondents. Obtaining consent from all respondents is a difficult challenge and one that faces significant resistance. Consequently, data linkage consent rates vary widely from study-to-study. Several studies have found significant differences between consenters and non-consenters on socio-demographic variables, but no study has investigated the underlying mechanisms of consent from a theory-driven perspective. In this study, we describe and test several hypotheses related to respondents' willingness to consent to an earnings and benefit data linkage request based on mechanisms related to financial uncertainty, privacy concerns, resistance towards the survey interview, level of attentiveness during the interview, the respondents' preexisting relationship with the administrative data agency, and matching respondents and interviewers on observable characteristics. The results point to several implications for survey practice and suggestions for future research.
RESUMEN
Ownership of a bank account is an objective measure and should be relatively easy to elicit via survey questions. Yet, depending on the interview mode, the wording of the question and its placement within the survey may influence respondents' answers. The Health and Retirement Study (HRS) asset module, as administered online to members of the Understanding America Study (UAS), yielded substantially lower rates of reported bank account ownership than either a single question on ownership in the Current Population Survey (CPS) or the full asset module administered to HRS panelists (both interviewer-administered surveys). We designed and implemented an experiment in the UAS comparing the original HRS question eliciting bank account ownership with two alternative versions that were progressively simplified. We document strong evidence that the original question leads to systematic underestimation of bank account ownership. In contrast, the proportion of bank account owners obtained from the simplest alternative version of the question is very similar to the population benchmark estimate. We investigate treatment effect heterogeneity by cognitive ability and financial literacy. We find that questionnaire simplification affects responses of individuals with higher cognitive ability substantially less than those with lower cognitive ability. Our results suggest that high-quality data from surveys start from asking the right questions, which should be as simple and precise as possible and carefully adapted to the mode of interview.
RESUMEN
Longitudinal surveys traditionally conducted by interviewers are facing increasing pressures to explore alternatives such as sequential mixed-mode designs, which start with a cheaper self-administered mode (online) then follow up using more expensive methods such as telephone or face-to-face interviewing. Using a designed experiment conducted as part of the 2018 wave of the Health and Retirement Study (HRS) in the US, we compare a sequential mixed-mode design (web then telephone) with the standard telephone-only protocol. Using an intent-to-treat analysis, we focus on response quality and response distributions for several domains key to HRS: physical and psychological health, financial status, expectations and family composition. Respondents assigned to the sequential mixed-mode (web) had slightly higher missing data rates and more focal responses than those assigned to telephone-only. However, we find no evidence of differential quality in verifying and updating roster information. We find slightly lower rates of asset ownership reported by those assigned to the web mode. Conditional on ownership, we find no detectable mode effects on the value of assets. We find more negative (pessimistic) expectations for those assigned to the web mode. We find little evidence of poorer health reported by those assigned to the web mode. We find that effects of mode assignment on measurement are present, but for most indicators the effects are small. Finding ways to remediate the differences in item-missing data and focal values should help reduce mode effects in mixed-mode surveys or those transitioning from interviewer- to self-administration.
Asunto(s)
Exactitud de los Datos , Teléfono , Encuestas y Cuestionarios , Estudios LongitudinalesRESUMEN
Introduction: Updating the mode of data collection may affect response rates or survey results. The ongoing, national Monitoring the Future (MTF) panel study has traditionally used mailed paper surveys. In 2018, MTF experimented with a web-push data collection design for young adults ages 19-30, concluding that the web-push design improved response rates and did not change substance use estimates after controlling for sociodemographic characteristics (Patrick et al., 2021). The current study sought to replicate the web-push experiment with MTF adults ages 35 to 60 in 2020. Methods: In 2020, the MTF panel study included an experiment to test a web-push protocol for respondents ages 35 to 60 (N = 14,379). Participants were randomized to the web-push (i.e., a web survey invitation, with paper surveys available for non-respondents) or traditional MTF (i.e., mailed paper surveys) data collection condition. Results: Results indicated no significant difference in overall response rate for the web-push vs. standard MTF conditions in this age group. Differences in reported estimates of past 30-day substance use prevalence by condition were not significant after adjusting for sociodemographic characteristics. In multivariable models, participants in the web-push condition were less likely to respond via web (than paper) if they were Black, smoked cigarettes in the past 30 days, were unmarried, or did not have a college degree. Conclusions: Overall, the move to the web-push design had minimal impact on response rates and substance use prevalence estimates for this age group. However, in the web-push condition, sociodemographic differences were associated with mode of response.