A Two-Step Framework for Validating Causal Effect Estimates.

BACKGROUND: Comparing causal effect estimates obtained using observational data to those obtained from the gold standard (i.e., randomized controlled trials [RCTs]) helps assess the validity of these estimates. However, comparisons are challenging due to differences between observational data and RCT generated data. The unknown treatment assignment mechanism in the observational data and varying sampling mechanisms between the RCT and the observational data can lead to confounding and sampling bias, respectively. AIMS: The objective of this study is to propose a two-step framework to validate causal effect estimates obtained from observational data by adjusting for both mechanisms. MATERIALS AND METHODS: An estimator of causal effects related to the two mechanisms is constructed. A two-step framework for comparing causal effect estimates is derived from the estimator. An R package RCTrep is developed to implement the framework in practice. RESULTS: A simulation study is conducted to show that using our framework observational data can produce causal effect estimates similar to those of an RCT. A real-world application of the framework to validate treatment effects of adjuvant chemotherapy obtained from registry data is demonstrated. CONCLUSION: This  study constructs a framework for comparing causal effect estimates between observational data and RCT data, facilitating the assessment of the validity of causal effect estimates obtained from observational data.

Bayesian Multivariate Logistic Regression for Superiority and Inferiority Decision-Making under Observable Treatment Heterogeneity.

The effects of treatments may differ between persons with different characteristics. Addressing such treatment heterogeneity is crucial to investigate whether patients with specific characteristics are likely to benefit from a new treatment. The current paper presents a novel Bayesian method for superiority decision-making in the context of randomized controlled trials with multivariate binary responses and heterogeneous treatment effects. The framework is based on three elements: a) Bayesian multivariate logistic regression analysis with a Pólya-Gamma expansion; b) a transformation procedure to transfer obtained regression coefficients to a more intuitive multivariate probability scale (i.e., success probabilities and the differences between them); and c) a compatible decision procedure for treatment comparison with prespecified decision error rates. Procedures for a priori sample size estimation under a non-informative prior distribution are included. A numerical evaluation demonstrated that decisions based on a priori sample size estimation resulted in anticipated error rates among the trial population as well as subpopulations. Further, average and conditional treatment effect parameters could be estimated unbiasedly when the sample was large enough. Illustration with the International Stroke Trial dataset revealed a trend toward heterogeneous effects among stroke patients: Something that would have remained undetected when analyses were limited to average treatment effects.

Bayesian multilevel multivariate logistic regression for superiority decision-making under observable treatment heterogeneity.

BACKGROUND: In medical, social, and behavioral research we often encounter datasets with a multilevel structure and multiple correlated dependent variables. These data are frequently collected from a study population that distinguishes several subpopulations with different (i.e., heterogeneous) effects of an intervention. Despite the frequent occurrence of such data, methods to analyze them are less common and researchers often resort to either ignoring the multilevel and/or heterogeneous structure, analyzing only a single dependent variable, or a combination of these. These analysis strategies are suboptimal: Ignoring multilevel structures inflates Type I error rates, while neglecting the multivariate or heterogeneous structure masks detailed insights. METHODS: To analyze such data comprehensively, the current paper presents a novel Bayesian multilevel multivariate logistic regression model. The clustered structure of multilevel data is taken into account, such that posterior inferences can be made with accurate error rates. Further, the model shares information between different subpopulations in the estimation of average and conditional average multivariate treatment effects. To facilitate interpretation, multivariate logistic regression parameters are transformed to posterior success probabilities and differences between them. RESULTS: A numerical evaluation compared our framework to less comprehensive alternatives and highlighted the need to model the multilevel structure: Treatment comparisons based on the multilevel model had targeted Type I error rates, while single-level alternatives resulted in inflated Type I errors. Further, the multilevel model was more powerful than a single-level model when the number of clusters was higher. A re-analysis of the Third International Stroke Trial data illustrated how incorporating a multilevel structure, assessing treatment heterogeneity, and combining dependent variables contributed to an in-depth understanding of treatment effects. Further, we demonstrated how Bayes factors can aid in the selection of a suitable model. CONCLUSION: The method is useful in prediction of treatment effects and decision-making within subpopulations from multiple clusters, while taking advantage of the size of the entire study sample and while properly incorporating the uncertainty in a principled probabilistic manner using the full posterior distribution.

Personalization in biomedical-informatics: Methodological considerations and recommendations.

Over the last decades there has been an increasing interest in personalization: can we make sure that treatments are effective for individual patients? The quest for personalization affects biomedical informatics in two ways: first, we design systems-for example eHealth applications-that directly interact with patients and these systems might themselves one day be personalized. Hence, we seek effective methods to do so. Second, we design systems that collect the data which will one day be used to personalize treatments: hence, we need to critically consider design requirements that improve the utility of (e.g.,) personal health records for future treatment personalization. By clearly defining personalization and analyzing the effectiveness of different personalization methods this discussion highlights how we should embrace sequential experimentation-as opposed to the traditional randomized trial-if we want to personalize our informatics systems efficiently. Furthermore, we need to make sure that we capture the treatment assignment process in our health records: doing so will greatly increase the utility of the collected data for future personalization attempts.

The use of Thompson sampling to increase estimation precision.

In this article, we consider a sequential sampling scheme for efficient estimation of the difference between the means of two independent treatments when the population variances are unequal across groups. The sampling scheme proposed is based on a solution to bandit problems called Thompson sampling. While this approach is most often used to maximize the cumulative payoff over competing treatments, we show that the same method can also be used to balance exploration and exploitation when the aim of the experimenter is to efficiently increase estimation precision. We introduce this novel design optimization method and, by simulation, show its effectiveness.

Value of Real-World Evidence for Treatment Selection: A Case Study in Colon Cancer.

PURPOSE: Real-world evidence (RWE)-derived from analysis of real-world data (RWD)-has the potential to guide personalized treatment decisions. However, because of potential confounding, generating valid RWE is challenging. This study demonstrates how to responsibly generate RWE for treatment decisions. We validate our approach by demonstrating that we can uncover an existing adjuvant chemotherapy (ACT) guideline for stage II and III colon cancer (CC)-which came about using both data from randomized controlled trials and expert consensus-solely using RWD. METHODS: Data from the population-based Netherlands Cancer Registry from a total of 27,056 patients with stage II and III CC who underwent curative surgery were analyzed to estimate the overall survival (OS) benefit of ACT. Focusing on 5-year OS, the benefit of ACT was estimated for each patient using G-computation methods by adjusting for patient and tumor characteristics and estimated propensity score. Subsequently, on the basis of these estimates, an ACT decision tree was constructed. RESULTS: The constructed decision tree corresponds to the current Dutch guideline: patients with stage III or stage II with T stage 4 should receive surgery and ACT, whereas patients with stage II with T stage 3 should only receive surgery. Interestingly, we do not find sufficient RWE to conclude against ACT for stage II with T stage 4 and microsatellite instability-high (MSI-H), a recent addition to the current guideline. CONCLUSION: RWE, if used carefully, can provide a valuable addition to our construction of evidence on clinical decision making and therefore ultimately affect treatment guidelines. Next to validating the ACT decisions advised in the current Dutch guideline, this paper suggests additional attention should be paid to MSI-H in future iterations of the guideline.

Process and Information Needs When Searching for and Selecting Apps for Smoking Cessation: Qualitative Study Using Contextual Inquiry.

BACKGROUND: Hundreds of apps are available to support people in their quest to quit smoking. It has been hypothesized that selecting an app from a sizable volume without any aid can be overwhelming and difficult. However, little is known about how people choose apps for smoking cessation and what exactly people want to know about an app before choosing to install it. Understanding the decision-making process may ultimately be helpful in creating tools to help people meaningfully select apps. OBJECTIVE: The aim of this study is to obtain insights into the process of searching and selecting mobile apps for smoking cessation and map the range of actions and the accompanying reasons during the search, focusing on the information needs and experiences of those who aim to find an app. METHODS: Contextual inquiries were conducted with 10 Dutch adults wanting to quit smoking by using an app. During the inquiries, we observed people as they chose an app. In addition, 2 weeks later, there was a short semistructured follow-up interview over the phone. Through convenience and purposive sampling, we included participants differing in gender, age, and educational level. We used thematic analysis to analyze the transcribed interviews and leveraged a combination of video and audio recordings to understand what is involved in searching and selecting apps for smoking cessation. RESULTS: The process of finding smoking cessation apps is comprehensive: participants explored, evaluated, and searched for information; imagined using functions; compared apps; assessed the trustworthiness of apps and information; and made several decisions while navigating the internet and app stores. During the search, the participants gained knowledge of apps and developed clearer ideas about their wishes and requirements. Confidence and trust in these apps to help quitting remained quite low or even decreased. Although the process was predominantly a positive experience, the whole process took time and energy and caused negative emotions such as frustration and disappointment for some participants. In addition, without the participants realizing it, errors in information processing occurred, which affected the choices they made. All participants chose an app with the explicit intention of using it. After 2 weeks, of the 10 participants, 6 had used the app, of whom only 1 extensively. CONCLUSIONS: Finding an app in the current app stores that contains functions and features expected to help in quitting smoking takes considerable time and energy, can be a negative experience, and is prone to errors in information processing that affect decision-making. Therefore, we advise the further development of decision aids, such as advanced filters, recommender systems and curated health app portals, and make a number of concrete recommendations for the design of such systems.

Decision-making with multiple correlated binary outcomes in clinical trials.

Clinical trials often evaluate multiple outcome variables to form a comprehensive picture of the effects of a new treatment. The resulting multidimensional insight contributes to clinically relevant and efficient decision-making about treatment superiority. Common statistical procedures to make these superiority decisions with multiple outcomes have two important shortcomings, however: (1) Outcome variables are often modeled individually, and consequently fail to consider the relation between outcomes; and (2) superiority is often defined as a relevant difference on a single, on any, or on all outcome(s); and lacks a compensatory mechanism that allows large positive effects on one or multiple outcome(s) to outweigh small negative effects on other outcomes. To address these shortcomings, this paper proposes (1) a Bayesian model for the analysis of correlated binary outcomes based on the multivariate Bernoulli distribution; and (2) a flexible decision criterion with a compensatory mechanism that captures the relative importance of the outcomes. A simulation study demonstrates that efficient and unbiased decisions can be made while Type I error rates are properly controlled. The performance of the framework is illustrated for (1) fixed, group sequential, and adaptive designs; and (2) non-informative and informative prior distributions.

Developing Effective Methods for Electronic Health Personalization: Protocol for Health Telescope, a Prospective Interventional Study.

BACKGROUND: Existing evaluations of the effects of mobile apps to encourage physical activity have been criticized owing to their common lack of external validity, their short duration, and their inability to explain the drivers of the observed effects. This protocol describes the setup of Health Telescope, a longitudinal panel study in which the long-term effects of mobile electronic health (eHealth) apps are investigated. By setting up Health Telescope, we aim to (1) understand more about the long-term use of eHealth apps in an externally valid setting, (2) understand the relationships between short-term and long-term outcomes of the usage of eHealth apps, and (3) test different ways in which eHealth app allocation can be personalized. OBJECTIVE: The objectives of this paper are to (1) demonstrate and motivate the validity of the many choices that we made in setting up an intensive longitudinal study, (2) provide a resource for researchers interested in using data generated by our study, and (3) act as a guideline for researchers interested in setting up their own longitudinal data collection using wearable devices. For the third objective, we explicitly discuss the General Data Protection Regulation and ethical requirements that need to be addressed. METHODS: In this 4-month study, a group of approximately 450 participants will have their daily step count measured and will be asked daily about their mood using experience sampling. Once per month, participants will receive an intervention containing a recommendation to download an app that focuses on increasing physical activity. The mechanism for assigning recommendations to participants will be personalized over time, using contextual data obtained from previous interventions. RESULTS: The data collection software has been developed, and all the legal and ethical checks are in place. Recruitment will start in Q4 of 2020. The initial results will be published in 2021. CONCLUSIONS: The aim of Health Telescope is to investigate how different individuals respond to different ways of being encouraged to increase their physical activity. In this paper, we detail the setup, methods, and analysis plan that will enable us to reach this aim. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): PRR1-10.2196/16471.

A practical approach to sample size calculation for fixed populations.

Researchers routinely compute desired sample sizes of clinical trials to control type-i and type-ii errors. While for many experimental designs sample size calculations are well-known, it remains an active area of research. Work in this area focusses predominantly on controlling properties of the trial. In this paper we provide ready-to-use methods to compute sample sizes using an alternative objective, namely that of maximizing the outcome for a whole population. Considering the expected outcome of both the trial, and the resulting guideline, we formulate and numerically analyze the expected value of the entire allocation procedure. Our approach strongly relates to theoretical work presented in the 60's which demonstrated the effectiveness of allocation procedures that incorporate population sizes when planning experiments over designs that focus solely on error rates within the trial. We add to this work by a) extending to alternative designs (mean comparisons not assuming equal variances and comparisons of proportions), b) providing easy-to-use software to compute sample sizes for multiple experimental designs, and c) presenting numerical analysis that demonstrate the efficiency of the suggested approach.

Uncovering noisy social signals: Using optimization methods from experimental physics to study social phenomena.

Due to the ubiquitous presence of treatment heterogeneity, measurement error, and contextual confounders, numerous social phenomena are hard to study. Precise control of treatment variables and possible confounders is often key to the success of studies in the social sciences, yet often proves out of the realm of control of the experimenter. To amend this situation we propose a novel approach coined "lock-in feedback" which is based on a method that is routinely used in high-precision physics experiments to extract small signals out of a noisy environment. Here, we adapt the method to noisy social signals in multiple dimensions and evaluate it by studying an inherently noisy topic: the perception of (subjective) beauty. We show that the lock-in feedback approach allows one to select optimal treatment levels despite the presence of considerable noise. Furthermore, through the introduction of an external contextual shock we demonstrate that we can find relationships between noisy variables that were hitherto unknown. We therefore argue that lock-in methods may provide a valuable addition to the social scientist's experimental toolbox and we explicitly discuss a number of future applications.