An evaluation of whether propensity score adjustment can remove the self-selection bias inherent to web panel surveys addressing sensitive health behaviours.

BACKGROUND: In health research, population estimates are generally obtained from probability-based surveys. In market research surveys are frequently conducted from volunteer web panels. Propensity score adjustment (PSA) is often used at analysis to try to remove bias in the web survey, but empirical evidence of its effectiveness is mixed. We assess the ability of PSA to remove bias in the context of sensitive sexual health research and the potential of web panel surveys to replace or supplement probability surveys. METHODS: Four web panel surveys asked a subset of questions from the third British National Survey of Sexual Attitudes and Lifestyles (Natsal-3). Five propensity scores were generated for each web survey. The scores were developed from progressively larger sets of variables, beginning with demographic variables only and ending with demographic, sexual identity, lifestyle, attitudinal and sexual behaviour variables together. The surveys were weighted to match Natsal-3 based on propensity score quintiles. The performance of each survey and weighting was assessed by calculating the average 'absolute' odds ratio (inverse of the odds ratio if less than 1) across 22 pre-specified sexual behaviour outcomes of interest comparing the weighted web survey with Natsal-3. The average standard error across odds ratios was examined to assess the impact of weighting upon variance. RESULTS: Propensity weighting reduced bias relative to Natsal-3 as more variables were added for males, but had little effect for females, and variance increased for some surveys. Surveys with more biased estimates before propensity weighting showed greater reduction in bias from adjustment. Inconsistencies in performance were evident across surveys and outcomes. For most surveys and outcomes any reduction in bias was only partial and for some outcomes the bias increased. CONCLUSIONS: Even after propensity weighting using a rich range of information, including some sexual behaviour variables, some bias remained and variance increased for some web surveys. Whilst our findings support the use of PSA for web panel surveys, the reduction in bias is likely to be partial and unpredictable, consistent with the findings from market research. Our results do not support the use of volunteer web panels to generate unbiased population health estimates.

Development of the Initial Surveys for the All of Us Research Program.

BACKGROUND: The All of Us Research Program is building a national longitudinal cohort and collecting data from multiple information sources (e.g., biospecimens, electronic health records, and mobile/wearable technologies) to advance precision medicine. Participant-provided information, collected via surveys, will complement and augment these information sources. We report the process used to develop and refine the initial three surveys for this program. METHODS: The All of Us survey development process included: (1) prioritization of domains for scientific needs, (2) examination of existing validated instruments, (3) content creation, (4) evaluation and refinement via cognitive interviews and online testing, (5) content review by key stakeholders, and (6) launch in the All of Us electronic participant portal. All content was translated into Spanish. RESULTS: We conducted cognitive interviews in English and Spanish with 169 participants, and 573 individuals completed online testing. Feedback led to over 40 item content changes. Lessons learned included: (1) validated survey instruments performed well in diverse populations reflective of All of Us; (2) parallel evaluation of multiple languages can ensure optimal survey deployment; (3) recruitment challenges in diverse populations required multiple strategies; and (4) key stakeholders improved integration of surveys into larger Program context. CONCLUSIONS: This efficient, iterative process led to successful testing, refinement, and launch of three All of Us surveys. Reuse of All of Us surveys, available at http://researchallofus.org, may facilitate large consortia targeting diverse populations in English and Spanish to capture participant-provided information to supplement other data, such as genetic, physical measurements, or data from electronic health records.

The utility of a follow-up interview for respondents to a longitudinal survey with frequent measurement.

Social processes that change quickly are difficult to study, because they require frequent survey measurement. Weekly, daily, or even hourly measurement may be needed depending on the topic. With more frequent measurement comes the prospect of more complex patterns of missing data. The mechanisms creating the missing data may be varied, ranging from technical issues such as lack of an Internet connection to refusal to complete a requested survey. We examine one approach to mitigating the damage of these missing data - a follow-up or closeout interview that is completed after the frequent measurement. The Relationship Dynamics and Social Life (RDSL) study used this approach. The study asked women weekly about their attitudes and behaviors related to sexual relationships and pregnancy. The surveys were carried out for 130 weeks and concluded with a closeout interview. We explore the patterns of missing data in the RDSL study and then examine associations between the data collected in the closeout survey and key variables collected in the weekly survey. We then assess the extent to which data from the closeout survey are useful in repairing the damage caused by missing data.

New options for national population surveys: The implications of internet and smartphone coverage.

Challenges to survey data collection have increased the costs of social research via face-to-face surveys so much that it may become extremely difficult for social scientists to continue using these methods. A key drawback to less expensive Internet-based alternatives is the threat of biased results from coverage errors in survey data. The rise of Internet-enabled smartphones presents an opportunity to re-examine the issue of Internet coverage for surveys and its implications for coverage bias. Two questions (on Internet access and smartphone ownership) were added to the National Survey of Family Growth (NSFG), a U.S. national probability survey of women and men age 15-44, using a continuous sample design. We examine 16 quarters (4 years) of data, from September 2012 to August 2016. Overall, we estimate that 82.9% of the target NSFG population has Internet access, and 81.6% has a smartphone. Combined, this means that about 90.7% of U.S. residents age 15-44 have Internet access, via either traditional devices or a smartphone. We find some evidence of compensatory coverage when looking at key race/ethnicity and age subgroups. For instance, while Black teens (15-18) have the lowest estimated rate of Internet access (81.9%) and the lowest rate of smartphone usage (72.6%), an estimated 88.0% of this subgroup has some form of Internet access. We also examine the socio-demographic correlates of Internet and smartphone coverage, separately and combined, as indicators of technology access in this population. In addition, we look at the effect of differential coverage on key estimates produced by the NSFG, related to fertility, family formation, and sexual activity. While this does not address nonresponse or measurement biases that may differ for alternative modes, our paper has implications for possible coverage biases that may arise when switching to a Web-based mode of data collection, either for follow-up surveys or to replace the main face-to-face data collection.

Response of sensitive behaviors to frequent measurement.

We study the influence of frequent survey measurement on behavior. Widespread access to the Internet has made important breakthroughs in frequent measurement possible-potentially revolutionizing social science measurement of processes that change quickly over time. One key concern about using such frequent measurement is that it may influence the behavior being studied. We investigate this possibility using both a population-based experiment with random assignment to participation in a weekly journal for twelve months (versus no journal) and a large-scale, population-based, journal-keeping study with weekly measurement for 30 months. Results reveal few of the measured behaviors are correlated with assignment to frequent measurement. Theoretical reasoning regarding the likely behavioral response to frequent measurement correctly predicts domains most vulnerable to this possibility. Overall, however, we found little evidence of behavioral response to frequent measurement.

Assessing the need for a new nationally representative household panel survey in the United States.

We introduce this special issue on the critical matter of whether the existing household panel surveys in the U.S. are adequate to address the important emerging social science and policy questions of the next few decades. We summarize the conference papers which address this issue in different domains. The papers detail many new and important emerging research questions but also identify key limitations in existing panels in addressing those questions. To address these limitations, we consider the advantages and disadvantages of initiating a new, general-purpose omnibus household panel in the U.S. We also discuss the particular benefits of starting new panels that have specific targeted domains such as child development, population health and health care. We also develop a list of valuable enhancements to existing panels which could address many of their limitations.

The role of perceived benefits and costs in patients' medical decisions.

BACKGROUND: Many decisions can be understood in terms of actors' valuations of benefits and costs. The article investigates whether this is also true of patient medical decision making. It aims to investigate (i) the importance patients attach to various reasons for and against nine medical decisions; (ii) how well the importance attached to benefits and costs predicts action or inaction; and (iii) how such valuations are related to decision confidence. METHODS: In a national random digit dial telephone survey of U.S. adults, patients rated the importance of various reasons for and against medical decisions they had made or talked to a health-care provider about during the past 2 years. Participants were 2575 English-speaking adults age 40 and older. Data were analysed by means of logistic regressions predicting action/inaction and linear regressions predicting confidence. RESULTS: Aggregating individual reasons into those that may be regarded as benefits and those that may be regarded as costs, and weighting them by their importance to the patient, shows the expected relationship to action. Perceived benefits and costs are also significantly related to the confidence patients report about their decision. CONCLUSION: The factors patients say are important in their medical decisions reflect a subjective weighing of benefits and costs and predict action/inaction although they do not necessarily indicate that patients are well informed. The greater the difference between the importance attached to benefits and costs, the greater patients' confidence in their decision.

Nonprobability Web surveys to measure sexual behaviors and attitudes in the general population: a comparison with a probability sample interview survey.

BACKGROUND: Nonprobability Web surveys using volunteer panels can provide a relatively cheap and quick alternative to traditional health and epidemiological surveys. However, concerns have been raised about their representativeness. OBJECTIVE: The aim was to compare results from different Web panels with a population-based probability sample survey (n=8969 aged 18-44 years) that used computer-assisted self-interview (CASI) for sensitive behaviors, the third British National Survey of Sexual Attitudes and Lifestyles (Natsal-3). METHODS: Natsal-3 questions were included on 4 nonprobability Web panel surveys (n=2000 to 2099), 2 using basic quotas based on age and sex, and 2 using modified quotas based on additional variables related to key estimates. Results for sociodemographic characteristics were compared with external benchmarks and for sexual behaviors and opinions with Natsal-3. Odds ratios (ORs) were used to express differences between the benchmark data and each survey for each variable of interest. A summary measure of survey performance was the average absolute OR across variables. Another summary measure was the number of key estimates for which the survey differed significantly (at the 5% level) from the benchmarks. RESULTS: For sociodemographic variables, the Web surveys were less representative of the general population than Natsal-3. For example, for men, the average absolute OR for Natsal-3 was 1.14, whereas for the Web surveys the average absolute ORs ranged from 1.86 to 2.30. For all Web surveys, approximately two-thirds of the key estimates of sexual behaviors were different from Natsal-3 and the average absolute ORs ranged from 1.32 to 1.98. Differences were appreciable even for questions asked by CASI in Natsal-3. No single Web survey performed consistently better than any other did. Modified quotas slightly improved results for men, but not for women. CONCLUSIONS: Consistent with studies from other countries on less sensitive topics, volunteer Web panels provided appreciably biased estimates. The differences seen with Natsal-3 CASI questions, where mode effects may be similar, suggest a selection bias in the Web surveys. The use of more complex quotas may lead to some improvement, but many estimates are still likely to differ. Volunteer Web panels are not recommended if accurate prevalence estimates for the general population are a key objective.

The Design of Grids in Web Surveys.

Grid or matrix questions are associated with a number of problems in Web surveys. In this paper, we present results from two experiments testing the design of grid questions to reduce breakoffs, missing data, and satisficing. The first examines dynamic elements to help guide respondent through the grid, and on splitting a larger grid into component pieces. The second manipulates the visual complexity of the grid and on simplifying the grid. We find that using dynamic feedback to guide respondents through a multi-question grid helps reduce missing data. Splitting the grids into component questions further reduces missing data and motivated underreporting. The visual complexity of the grid appeared to have little effect on performance.

Visible Cash, a Second Incentive, and Priority Mail? An Experimental Evaluation of Mailing Strategies for a Screening Questionnaire in a National Push-to-Web/Mail Survey.

In push-to-web surveys that use postal mail to contact sampled cases, participation is contingent on the mail being opened and the survey invitations being delivered. The design of the mailings is crucial to the success of the survey. We address the question of how to design invitation mailings that can grab potential respondents' attention and sway them to be interested in the survey in a short window of time. In the household screening stage of a national survey, the American Family Health Study, we experimentally tested three mailing design techniques for recruiting respondents: (1) a visible cash incentive in the initial mailing, (2) a second incentive for initial nonrespondents, and (3) use of Priority Mail in the nonresponse follow-up mailing. We evaluated the three techniques' overall effects on response rates as well as how they differentially attracted respondents with different characteristics. We found that all three techniques were useful in increasing the screening response rates, but there was little evidence that they had differential effects on sample subgroups that could help to reduce nonresponse biases.

An Experimental Evaluation of Two Approaches for Improving Response to Household Screening Efforts in National Mail/Web Surveys.

Survey researchers have carefully modified their data collection operations for various reasons, including the rising costs of data collection and the ongoing Coronavirus disease (COVID-19) pandemic, both of which have made in-person interviewing difficult. For large national surveys that require household (HH) screening to determine survey eligibility, cost-efficient screening methods that do not include in-person visits need additional evaluation and testing. A new study, known as the American Family Health Study (AFHS), recently initiated data collection with a national probability sample, using a sequential mixed-mode mail/web protocol for push-to-web US HH screening (targeting persons aged 18-49 years). To better understand optimal approaches for this type of national screening effort, we embedded two randomized experiments in the AFHS data collection. The first tested the use of bilingual respondent materials where mailed invitations to the screener were sent in both English and Spanish to 50 percent of addresses with a high predicted likelihood of having a Spanish speaker and 10 percent of all other addresses. We found that the bilingual approach did not increase the response rate of high-likelihood Spanish-speaking addresses, but consistent with prior work, it increased the proportion of eligible Hispanic respondents identified among completed screeners, especially among addresses predicted to have a high likelihood of having Spanish speakers. The second tested a form of nonresponse follow-up, where a subsample of active sampled HHs that had not yet responded to the screening invitations was sent a priority mailing with a $5 incentive, adding to the $2 incentive provided for all sampled HHs in the initial screening invitation. We found this approach to be quite valuable for increasing the screening survey response rate.

Shedding light on participant selection bias in Ecological Momentary Assessment (EMA) studies: Findings from an internet panel study.

Although the potential for participant selection bias is readily acknowledged in the momentary data collection literature, very little is known about uptake rates in these studies or about differences in the people that participate versus those who do not. This study analyzed data from an existing Internet panel of older people (age 50 and greater) who were offered participation into a momentary study (n = 3,169), which made it possible to compute uptake and to compare many characteristics of participation status. Momentary studies present participants with brief surveys multiple times a day over several days; these surveys ask about immediate or recent experiences. A 29.1% uptake rate was observed when all respondents were considered, whereas a 39.2% uptake rate was found when individuals who did not have eligible smartphones (necessary for ambulatory data collection) were eliminated from the analyses. Taking into account the participation rate for being in this Internet panel, we estimate uptake rates for the general population to be about 5%. A consistent pattern of differences emerged between those who accepted the invitation to participate versus those who did not (in univariate analyses): participants were more likely to be female, younger, have higher income, have higher levels of education, rate their health as better, be employed, not be retired, not be disabled, have better self-rated computer skills, and to have participated in more prior Internet surveys (all p < .0026). Many variables were not associated with uptake including race, big five personality scores, and subjective well-being. For several of the predictors, the magnitude of the effects on uptake was substantial. These results indicate the possibility that, depending upon the associations being investigated, person selection bias could be present in momentary data collection studies.

Importance of missingness in baseline variables: A case study of the All of Us Research Program.

OBJECTIVE: The All of Us Research Program collects data from multiple information sources, including health surveys, to build a national longitudinal research repository that researchers can use to advance precision medicine. Missing survey responses pose challenges to study conclusions. We describe missingness in All of Us baseline surveys. STUDY DESIGN AND SETTING: We extracted survey responses between May 31, 2017, to September 30, 2020. Missing percentages for groups historically underrepresented in biomedical research were compared to represented groups. Associations of missing percentages with age, health literacy score, and survey completion date were evaluated. We used negative binomial regression to evaluate participant characteristics on the number of missed questions out of the total eligible questions for each participant. RESULTS: The dataset analyzed contained data for 334,183 participants who submitted at least one baseline survey. Almost all (97.0%) of the participants completed all baseline surveys, and only 541 (0.2%) participants skipped all questions in at least one of the baseline surveys. The median skip rate was 5.0% of the questions, with an interquartile range (IQR) of 2.5% to 7.9%. Historically underrepresented groups were associated with higher missingness (incidence rate ratio (IRR) [95% CI]: 1.26 [1.25, 1.27] for Black/African American compared to White). Missing percentages were similar by survey completion date, participant age, and health literacy score. Skipping specific questions were associated with higher missingness (IRRs [95% CI]: 1.39 [1.38, 1.40] for skipping income, 1.92 [1.89, 1.95] for skipping education, 2.19 [2.09-2.30] for skipping sexual and gender questions). CONCLUSION: Surveys in the All of Us Research Program will form an essential component of the data researchers can use to perform their analyses. Missingness was low in All of Us baseline surveys, but group differences exist. Additional statistical methods and careful analysis of surveys could help mitigate challenges to the validity of conclusions.

LINKING SURVEY AND ADMINISTRATIVE RECORDS: MECHANISMS OF CONSENT.

Survey records are increasingly being linked to administrative databases to enhance the survey data and increase research opportunities for data users. A necessary prerequisite to linking survey and administrative records is obtaining informed consent from respondents. Obtaining consent from all respondents is a difficult challenge and one that faces significant resistance. Consequently, data linkage consent rates vary widely from study-to-study. Several studies have found significant differences between consenters and non-consenters on socio-demographic variables, but no study has investigated the underlying mechanisms of consent from a theory-driven perspective. In this study, we describe and test several hypotheses related to respondents' willingness to consent to an earnings and benefit data linkage request based on mechanisms related to financial uncertainty, privacy concerns, resistance towards the survey interview, level of attentiveness during the interview, the respondents' preexisting relationship with the administrative data agency, and matching respondents and interviewers on observable characteristics. The results point to several implications for survey practice and suggestions for future research.

A Simple Question Goes a Long Way: A Wording Experiment on Bank Account Ownership.

Ownership of a bank account is an objective measure and should be relatively easy to elicit via survey questions. Yet, depending on the interview mode, the wording of the question and its placement within the survey may influence respondents' answers. The Health and Retirement Study (HRS) asset module, as administered online to members of the Understanding America Study (UAS), yielded substantially lower rates of reported bank account ownership than either a single question on ownership in the Current Population Survey (CPS) or the full asset module administered to HRS panelists (both interviewer-administered surveys). We designed and implemented an experiment in the UAS comparing the original HRS question eliciting bank account ownership with two alternative versions that were progressively simplified. We document strong evidence that the original question leads to systematic underestimation of bank account ownership. In contrast, the proportion of bank account owners obtained from the simplest alternative version of the question is very similar to the population benchmark estimate. We investigate treatment effect heterogeneity by cognitive ability and financial literacy. We find that questionnaire simplification affects responses of individuals with higher cognitive ability substantially less than those with lower cognitive ability. Our results suggest that high-quality data from surveys start from asking the right questions, which should be as simple and precise as possible and carefully adapted to the mode of interview.

Data quality and response distributions in a mixed-mode survey.

Longitudinal surveys traditionally conducted by interviewers are facing increasing pressures to explore alternatives such as sequential mixed-mode designs, which start with a cheaper self-administered mode (online) then follow up using more expensive methods such as telephone or face-to-face interviewing. Using a designed experiment conducted as part of the 2018 wave of the Health and Retirement Study (HRS) in the US, we compare a sequential mixed-mode design (web then telephone) with the standard telephone-only protocol. Using an intent-to-treat analysis, we focus on response quality and response distributions for several domains key to HRS: physical and psychological health, financial status, expectations and family composition. Respondents assigned to the sequential mixed-mode (web) had slightly higher missing data rates and more focal responses than those assigned to telephone-only. However, we find no evidence of differential quality in verifying and updating roster information. We find slightly lower rates of asset ownership reported by those assigned to the web mode. Conditional on ownership, we find no detectable mode effects on the value of assets. We find more negative (pessimistic) expectations for those assigned to the web mode. We find little evidence of poorer health reported by those assigned to the web mode. We find that effects of mode assignment on measurement are present, but for most indicators the effects are small. Finding ways to remediate the differences in item-missing data and focal values should help reduce mode effects in mixed-mode surveys or those transitioning from interviewer- to self-administration.

A Web-Based Event History Calendar Approach for Measuring Contraceptive Use Behavior.

Event history calendars (EHCs) are frequently used in social measurement to capture important information about the time ordering of events in people's lives, and enable inference about the relationships of the events with other outcomes of interest. To date, EHCs have primarily been designed for face-to-face or telephone survey interviewing, and few calendar tools have been developed for more private, self-administered modes of data collection. Web surveys offer benefits in terms of both self-administration, which can reduce social desirability bias, and timeliness. We developed and tested a web application enabling the calendar-based measurement of contraceptive method use histories. These measures provide valuable information for researchers studying family planning and fertility behaviors. This study describes the development of the web application, and presents a comparison of data collected from online panels using the application with data from a benchmark face-to-face survey collecting similar measures (the National Survey of Family Growth).

Comparison of a web-push vs. mailed survey protocol in the Monitoring the Future panel study among adults ages 35 to 60.

Introduction: Updating the mode of data collection may affect response rates or survey results. The ongoing, national Monitoring the Future (MTF) panel study has traditionally used mailed paper surveys. In 2018, MTF experimented with a web-push data collection design for young adults ages 19-30, concluding that the web-push design improved response rates and did not change substance use estimates after controlling for sociodemographic characteristics (Patrick et al., 2021). The current study sought to replicate the web-push experiment with MTF adults ages 35 to 60 in 2020. Methods: In 2020, the MTF panel study included an experiment to test a web-push protocol for respondents ages 35 to 60 (N = 14,379). Participants were randomized to the web-push (i.e., a web survey invitation, with paper surveys available for non-respondents) or traditional MTF (i.e., mailed paper surveys) data collection condition. Results: Results indicated no significant difference in overall response rate for the web-push vs. standard MTF conditions in this age group. Differences in reported estimates of past 30-day substance use prevalence by condition were not significant after adjusting for sociodemographic characteristics. In multivariable models, participants in the web-push condition were less likely to respond via web (than paper) if they were Black, smoked cigarettes in the past 30 days, were unmarried, or did not have a college degree. Conclusions: Overall, the move to the web-push design had minimal impact on response rates and substance use prevalence estimates for this age group. However, in the web-push condition, sociodemographic differences were associated with mode of response.

Building on a Sequential Mixed-Mode Research Design in the Monitoring the Future Study.

Given the promise of the web push plus e-mail survey design for providing cost-effective and high-quality data (Patrick et al. 2018, 2019) as an alternative to a paper-and-pencil mailed survey design for the longitudinal Monitoring the Future (MTF) study, the current study sought to further enhance the web push condition. The MTF sample is based on US nationally representative samples of 12th grade students surveyed annually. The MTF control group for the current study included participants who completed the in-school baseline survey in the 12th grade and were selected to participate in their first follow-up survey in 2017 via mailed surveys (N = 1,222). A supplementary sample (N = ∼2,450) was assigned to one of the two sequential mixed-mode conditions. Those in condition 1 (N = 1,198), or mail push, were invited to complete mailed surveys and later given a web survey option. Those in condition 2 (N = 1,173), or enhanced web push, were invited to complete a web survey (the same as in the 2014 study, but with the addition of text messages and quick response (QR) codes and the web survey was optimized for mobile devices) and then later given a mailed survey option. Research aims were to examine response rates across conditions, as well as how responses were distributed across mode (paper, web), devices (computer, smartphone, table), and method of accessing the web survey (hand-entered URL, QR code, e-mail link, SMS link). Response rates differed significantly: the MTF control group was 34.2 percent, mail push was 35.4 percent, and enhanced web push was 42.05 percent. The higher response rate in the enhanced web push condition suggests that the additional strategies were effective at bringing in more respondents. Key estimates produced by the enhanced web push condition did not differ from those of the MTF control group.