A causal roadmap for generating high-quality real-world evidence.

Increasing emphasis on the use of real-world evidence (RWE) to support clinical policy and regulatory decision-making has led to a proliferation of guidance, advice, and frameworks from regulatory agencies, academia, professional societies, and industry. A broad spectrum of studies use real-world data (RWD) to produce RWE, ranging from randomized trials with outcomes assessed using RWD to fully observational studies. Yet, many proposals for generating RWE lack sufficient detail, and many analyses of RWD suffer from implausible assumptions, other methodological flaws, or inappropriate interpretations. The Causal Roadmap is an explicit, itemized, iterative process that guides investigators to prespecify study design and analysis plans; it addresses a wide range of guidance within a single framework. By supporting the transparent evaluation of causal assumptions and facilitating objective comparisons of design and analysis choices based on prespecified criteria, the Roadmap can help investigators to evaluate the quality of evidence that a given study is likely to produce, specify a study to generate high-quality RWE, and communicate effectively with regulatory agencies and other stakeholders. This paper aims to disseminate and extend the Causal Roadmap framework for use by clinical and translational researchers; three companion papers demonstrate applications of the Causal Roadmap for specific use cases.

Sensitivity analysis for causality in observational studies for regulatory science.

Objective: The United States Congress passed the 21st Century Cures Act mandating the development of Food and Drug Administration guidance on regulatory use of real-world evidence. The Forum on the Integration of Observational and Randomized Data conducted a meeting with various stakeholder groups to build consensus around best practices for the use of real-world data (RWD) to support regulatory science. Our companion paper describes in detail the context and discussion of the meeting, which includes a recommendation to use a causal roadmap for study designs using RWD. This article discusses one step of the roadmap: the specification of a sensitivity analysis for testing robustness to violations of causal model assumptions. Methods: We present an example of a sensitivity analysis from a RWD study on the effectiveness of Nifurtimox in treating Chagas disease, and an overview of various methods, emphasizing practical considerations on their use for regulatory purposes. Results: Sensitivity analyses must be accompanied by careful design of other aspects of the causal roadmap. Their prespecification is crucial to avoid wrong conclusions due to researcher degrees of freedom. Sensitivity analysis methods require auxiliary information to produce meaningful conclusions; it is important that they have at least two properties: the validity of the conclusions does not rely on unverifiable assumptions, and the auxiliary information required by the method is learnable from the corpus of current scientific knowledge. Conclusions: Prespecified and assumption-lean sensitivity analyses are a crucial tool that can strengthen the validity and trustworthiness of effectiveness conclusions for regulatory science.

Causal Inference in the Presence of Interference in Sponsored Search Advertising.

In classical causal inference, inferring cause-effect relations from data relies on the assumption that units are independent and identically distributed. This assumption is violated in settings where units are related through a network of dependencies. An example of such a setting is ad placement in sponsored search advertising, where the likelihood of a user clicking on a particular ad is potentially influenced by where it is placed and where other ads are placed on the search result page. In such scenarios, confounding arises due to not only the individual ad-level covariates but also the placements and covariates of other ads in the system. In this paper, we leverage the language of causal inference in the presence of interference to model interactions among the ads. Quantification of such interactions allows us to better understand the click behavior of users, which in turn impacts the revenue of the host search engine and enhances user satisfaction. We illustrate the utility of our formalization through experiments carried out on the ad placement system of the Bing search engine.

Population-scale dietary interests during the COVID-19 pandemic.

The SARS-CoV-2 virus has altered people's lives around the world. Here we document population-wide shifts in dietary interests in 18 countries in 2020, as revealed through time series of Google search volumes. We find that during the first wave of the COVID-19 pandemic there was an overall surge in food interest, larger and longer-lasting than the surge during typical end-of-year holidays in Western countries. The shock of decreased mobility manifested as a drastic increase in interest in consuming food at home and a corresponding decrease in consuming food outside of home. The largest (up to threefold) increases occurred for calorie-dense carbohydrate-based foods such as pastries, bakery products, bread, and pies. The observed shifts in dietary interests have the potential to globally affect food consumption and health outcomes. These findings can inform governmental and organizational decisions regarding measures to mitigate the effects of the COVID-19 pandemic on diet and nutrition.

Measuring algorithmically infused societies.

It has been the historic responsibility of the social sciences to investigate human societies. Fulfilling this responsibility requires social theories, measurement models and social data. Most existing theories and measurement models in the social sciences were not developed with the deep societal reach of algorithms in mind. The emergence of 'algorithmically infused societies'-societies whose very fabric is co-shaped by algorithmic and human behaviour-raises three key challenges: the insufficient quality of measurements, the complex consequences of (mis)measurements, and the limits of existing social theories. Here we argue that tackling these challenges requires new social theories that account for the impact of algorithmic systems on social realities. To develop such theories, we need new methodologies for integrating data and measurements into theory construction. Given the scale at which measurements can be applied, we believe measurement models should be trustworthy, auditable and just. To achieve this, the development of measurements should be transparent and participatory, and include mechanisms to ensure measurement quality and identify possible harms. We argue that computational social scientists should rethink what aspects of algorithmically infused societies should be measured, how they should be measured, and the consequences of doing so.

Understanding Side Effects of Antidepressants: Large-scale Longitudinal Study on Social Media Data.

BACKGROUND: Antidepressants are known to show heterogeneous effects across individuals and conditions, posing challenges to understanding their efficacy in mental health treatment. Social media platforms enable individuals to share their day-to-day concerns with others and thereby can function as unobtrusive, large-scale, and naturalistic data sources to study the longitudinal behavior of individuals taking antidepressants. OBJECTIVE: We aim to understand the side effects of antidepressants from naturalistic expressions of individuals on social media. METHODS: On a large-scale Twitter data set of individuals who self-reported using antidepressants, a quasi-experimental study using unsupervised language analysis was conducted to extract keywords that distinguish individuals who improved and who did not improve following the use of antidepressants. The net data set consists of over 8 million Twitter posts made by over 300,000 users in a 4-year period between January 1, 2014, and February 15, 2018. RESULTS: Five major side effects of antidepressants were studied: sleep, weight, eating, pain, and sexual issues. Social media language revealed keywords related to these side effects. In particular, antidepressants were found to show a spectrum of effects from decrease to increase in each of these side effects. CONCLUSIONS: This work enhances the understanding of the side effects of antidepressants by identifying distinct linguistic markers in the longitudinal social media data of individuals showing the most and least improvement following the self-reported intake of antidepressants. One implication of this work concerns the potential of social media data as an effective means to support digital pharmacovigilance and digital therapeutics. These results can inform clinicians in tailoring their discussion and assessment of side effects and inform patients about what to potentially expect and what may or may not be within the realm of normal aftereffects of antidepressants.

A Social Media Study on the Effects of Psychiatric Medication Use.

Understanding the effects of psychiatric medications during mental health treatment constitutes an active area of inquiry. While clinical trials help evaluate the effects of these medications, many trials suffer from a lack of generalizability to broader populations. We leverage social media data to examine psychopathological effects subject to self-reported usage of psychiatric medication. Using a list of common approved and regulated psychiatric drugs and a Twitter dataset of 300M posts from 30K individuals, we develop machine learning models to first assess effects relating to mood, cognition, depression, anxiety, psychosis, and suicidal ideation. Then, based on a stratified propensity score based causal analysis, we observe that use of specific drugs are associated with characteristic changes in an individual's psychopathology. We situate these observations in the psychiatry literature, with a deeper analysis of pre-treatment cues that predict treatment outcomes. Our work bears potential to inspire novel clinical investigations and to build tools for digital therapeutics.

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries.

Social data in digital form-including user-generated content, expressed or implicit relations between people, and behavioral traces-are at the core of popular applications and platforms, driving the research agenda of many researchers. The promises of social data are many, including understanding "what the world thinks" about a social issue, brand, celebrity, or other entity, as well as enabling better decision-making in a variety of fields including public policy, healthcare, and economics. Many academics and practitioners have warned against the naïve usage of social data. There are biases and inaccuracies occurring at the source of the data, but also introduced during processing. There are methodological limitations and pitfalls, as well as ethical boundaries and unexpected consequences that are often overlooked. This paper recognizes the rigor with which these issues are addressed by different researchers varies across a wide range. We identify a variety of menaces in the practices around social data use, and organize them in a framework that helps to identify them. "For your own sanity, you have to remember that not all problems can be solved. Not all problems can be solved, but all problems can be illuminated." -Ursula Franklin.

Geographically Resolved Rhythms in Twitter Use Reveal Social Pressures on Daily Activity Patterns.

Daily rhythms in human physiology and behavior are driven by the interplay of circadian rhythms, environmental cycles, and social schedules. Much research has focused on the mechanism and function of circadian rhythms in constant conditions or in idealized light-dark environments. There have been comparatively few studies into how social pressures, such as work and school schedules, affect human activity rhythms day to day and season to season. To address this issue, we analyzed activity on Twitter in >1,500 US counties throughout the 2012-2013 calendar years in 15-min intervals using geographically tagged tweets representing ≈0.1% of the total population each day. We find that sustained periods of low Twitter activity are correlated with sufficient sleep as measured by conventional surveys. We show that this nighttime lull in Twitter activity is shifted to later times on weekends relative to weekdays, a phenomenon we term "Twitter social jet lag." The magnitude of this social jet lag varies seasonally and geographically-with the West Coast experiencing less Twitter social jet lag compared to the Central and Eastern US-and is correlated with average commuting schedules and disease risk factors such as obesity. Most counties experience the largest amount of Twitter social jet lag in February and the lowest in June or July. We present evidence that these shifts in weekday activity coincide with relaxed social pressures due to local K-12 school holidays and that the direct seasonal effect of altered day length is comparatively weaker.

Conjunction of factors triggering waves of seasonal influenza.

Using several longitudinal datasets describing putative factors affecting influenza incidence and clinical data on the disease and health status of over 150 million human subjects observed over a decade, we investigated the source and the mechanistic triggers of influenza epidemics. We conclude that the initiation of a pan-continental influenza wave emerges from the simultaneous realization of a complex set of conditions. The strongest predictor groups are as follows, ranked by importance: (1) the host population's socio- and ethno-demographic properties; (2) weather variables pertaining to specific humidity, temperature, and solar radiation; (3) the virus' antigenic drift over time; (4) the host population'€™s land-based travel habits, and; (5) recent spatio-temporal dynamics, as reflected in the influenza wave auto-correlation. The models we infer are demonstrably predictive (area under the Receiver Operating Characteristic curve 80%) when tested with out-of-sample data, opening the door to the potential formulation of new population-level intervention and mitigation policies.

The Language of Social Support in Social Media and its Effect on Suicidal Ideation Risk.

Online social support is known to play a significant role in mental well-being. However, current research is limited in its ability to quantify this link. Challenges exist due to the paucity of longitudinal, pre- and post mental illness risk data, and reliable methods that can examine causality between past availability of support and future risk. In this paper, we propose a method to measure how the language of comments in Reddit mental health communities influences risk to suicidal ideation in the future. Incorporating human assessments in a stratified propensity score analysis based framework, we identify comparable subpopulations of individuals and measure the effect of online social support language. We interpret these linguistic cues with an established theoretical model of social support, and find that esteem and network support play a more prominent role in reducing forthcoming risk. We discuss the implications of our work for designing tools that can improve support provisions in online communities.

Online and Social Media Data As an Imperfect Continuous Panel Survey.

There is a large body of research on utilizing online activity as a survey of political opinion to predict real world election outcomes. There is considerably less work, however, on using this data to understand topic-specific interest and opinion amongst the general population and specific demographic subgroups, as currently measured by relatively expensive surveys. Here we investigate this possibility by studying a full census of all Twitter activity during the 2012 election cycle along with the comprehensive search history of a large panel of Internet users during the same period, highlighting the challenges in interpreting online and social media activity as the results of a survey. As noted in existing work, the online population is a non-representative sample of the offline world (e.g., the U.S. voting population). We extend this work to show how demographic skew and user participation is non-stationary and difficult to predict over time. In addition, the nature of user contributions varies substantially around important events. Furthermore, we note subtle problems in mapping what people are sharing or consuming online to specific sentiment or opinion measures around a particular topic. We provide a framework, built around considering this data as an imperfect continuous panel survey, for addressing these issues so that meaningful insight about public interest and opinion can be reliably extracted from online and social media data.

Discovering Shifts to Suicidal Ideation from Mental Health Content in Social Media.

History of mental illness is a major factor behind suicide risk and ideation. However research efforts toward characterizing and forecasting this risk is limited due to the paucity of information regarding suicide ideation, exacerbated by the stigma of mental illness. This paper fills gaps in the literature by developing a statistical methodology to infer which individuals could undergo transitions from mental health discourse to suicidal ideation. We utilize semi-anonymous support communities on Reddit as unobtrusive data sources to infer the likelihood of these shifts. We develop language and interactional measures for this purpose, as well as a propensity score matching based statistical approach. Our approach allows us to derive distinct markers of shifts to suicidal ideation. These markers can be modeled in a prediction framework to identify individuals likely to engage in suicidal ideation in the future. We discuss societal and ethical implications of this research.

Detecting application-level failures in component-based internet services.

Most Internet services (e-commerce, search engines, etc.) suffer faults. Quickly detecting these faults can be the largest bottleneck in improving availability of the system. We present Pinpoint, a methodology for automating fault detection in Internet services by: 1) observing low-level internal structural behaviors of the service; 2) modeling the majority behavior of the system as correct; and 3) detecting anomalies in these behaviors as possible symptoms of failures. Without requiring any a priori application-specific information, Pinpoint correctly detected 89%-96% of major failures in our experiments, as compared with 20%-70% detected by current application-generic techniques.