Selective prediction-set models with coverage rate guarantees.

The current approach to using machine learning (ML) algorithms in healthcare is to either require clinician oversight for every use case or use their predictions without any human oversight. We explore a middle ground that lets ML algorithms abstain from making a prediction to simultaneously improve their reliability and reduce the burden placed on human experts. To this end, we present a general penalized loss minimization framework for training selective prediction-set (SPS) models, which choose to either output a prediction set or abstain. The resulting models abstain when the outcome is difficult to predict accurately, such as on subjects who are too different from the training data, and achieve higher accuracy on those they do give predictions for. We then introduce a model-agnostic, statistical inference procedure for the coverage rate of an SPS model that ensembles individual models trained using K-fold cross-validation. We find that SPS ensembles attain prediction-set coverage rates closer to the nominal level and have narrower confidence intervals for its marginal coverage rate. We apply our method to train neural networks that abstain more for out-of-sample images on the MNIST digit prediction task and achieve higher predictive accuracy for ICU patients compared to existing approaches.

Estimating survival parameters under conditionally independent left truncation.

Databases derived from electronic health records (EHRs) are commonly subject to left truncation, a type of selection bias that occurs when patients need to survive long enough to satisfy certain entry criteria. Standard methods to adjust for left truncation bias rely on an assumption of marginal independence between entry and survival times, which may not always be satisfied in practice. In this work, we examine how a weaker assumption of conditional independence can result in unbiased estimation of common statistical parameters. In particular, we show the estimability of conditional parameters in a truncated dataset, and of marginal parameters that leverage reference data containing non-truncated data on confounders. The latter is complementary to observational causal inference methodology applied to real-world external comparators, which is a common use case for real-world databases. We implement our proposed methods in simulation studies, demonstrating unbiased estimation and valid statistical inference. We also illustrate estimation of a survival distribution under conditionally independent left truncation in a real-world clinico-genomic database.

Maximizing success in single-session EUS-directed transgastric ERCP: a retrospective cohort study to identify predictive factors of stent migration.

BACKGROUND AND AIMS: EUS-directed transgastric ERCP (the EDGE procedure) is a simplified method of performing ERCP in Roux-en-Y gastric bypass patients. The EDGE procedure involves placement of a lumen-apposing metal stent (LAMS) into the excluded stomach to serve as a conduit for passage of the duodenoscope for pancreatobiliary intervention. Originally a multistep process, urgent indications for ERCP have led to the development of single-session EDGE (SS-EDGE) with LAMS placement and ERCP performed in the same session. The goal of this study was to identify predictive factors of intraprocedural LAMS migration in SS-EDGE. METHODS: We conducted a multicenter retrospective review that included 9 tertiary medical centers across the United States. Data were collected and analyzed from 128 SS-EDGE procedures. The primary outcome was intraprocedural LAMS migration. Secondary outcomes were other procedural adverse events such as bleeding and perforation. RESULTS: Eleven LAMS migrations were observed in 128 procedures (8.6%). Univariate analysis of clinically relevant variables was performed, as was a binary logistic regression analysis of stent diameter and stent dilation. This revealed that use of a smaller (15 mm) diameter LAMS was an independent predictor of intraprocedural stent migration (odds ratio, 5.36; 95% confidence interval, 1.29-22.24; P = .021). Adverse events included 3 patients who required surgery and 2 who experienced intraprocedural bleeding. CONCLUSIONS: Use of a larger-diameter LAMS is a predictive factor for a nonmigrated stent and improved procedural success in SS-EDGE. Although larger patient cohorts are needed to adequately assess these findings, performance of LAMS dilation and fixation may also decrease risk of intraprocedural LAMS migration and improve procedural success.

Bayesian additional evidence for decision making under small sample uncertainty.

BACKGROUND: Statistical inference based on small datasets, commonly found in precision oncology, is subject to low power and high uncertainty. In these settings, drawing strong conclusions about future research utility is difficult when using standard inferential measures. It is therefore important to better quantify the uncertainty associated with both significant and non-significant results based on small sample sizes. METHODS: We developed a new method, Bayesian Additional Evidence (BAE), that determines (1) how much additional supportive evidence is needed for a non-significant result to reach Bayesian posterior credibility, or (2) how much additional opposing evidence is needed to render a significant result non-credible. Although based in Bayesian analysis, a prior distribution is not needed; instead, the tipping point output is compared to reasonable effect ranges to draw conclusions. We demonstrate our approach in a comparative effectiveness analysis comparing two treatments in a real world biomarker-defined cohort, and provide guidelines for how to apply BAE in practice. RESULTS: Our initial comparative effectiveness analysis results in a hazard ratio of 0.31 with 95% confidence interval (0.09, 1.1). Applying BAE to this result yields a tipping point of 0.54; thus, an observed hazard ratio of 0.54 or smaller in a replication study would result in posterior credibility for the treatment association. Given that effect sizes in this range are not extreme, and that supportive evidence exists from a similar published study, we conclude that this problem is worthy of further research. CONCLUSIONS: Our proposed method provides a useful framework for interpreting analytic results from small datasets. This can assist researchers in deciding how to interpret and continue their investigations based on an initial analysis that has high uncertainty. Although we illustrated its use in estimating parameters based on time-to-event outcomes, BAE easily applies to any normally-distributed estimator, such as those used for analyzing binary or continuous outcomes.

Fast permutation tests and related methods, for association between rare variants and binary outcomes.

In large-scale genetic studies, a primary aim is to test for an association between genetic variants and a disease outcome. The variants of interest are often rare and appear with low frequency among subjects. In this situation, statistical tests based on standard asymptotic results do not adequately control the type I error rate, especially if the case : control ratio is unbalanced. In this article, we propose the use of permutation and approximate unconditional tests for testing association with rare variants. We use novel analytical calculations to efficiently approximate the true type I error rate under common study designs, and in numerical studies show that the proposed classes of tests significantly improve upon standard testing methods. We also illustrate our methods in data from a recent case-control study for genetic causes of a severe side effect of a common drug treatment.

Broadening Eligibility Criteria and Diversity among Patients for Cancer Clinical Trials.

BACKGROUND: Certain populations have been historically underrepresented in clinical trials. Broadening eligibility criteria is one approach to inclusive clinical research and achieving enrollment goals. How broadened trial eligibility criteria affect the diversity of eligible participants is unknown. METHODS: Using a nationwide electronic health record-derived deidentified database, we identified a retrospective cohort of patients diagnosed with 22 cancer types between April 1, 2013 and December 31, 2022 who received systemic therapy (N=235,234) for cancer. We evaluated strict versus broadened eligibility criteria using performance status and liver, kidney, and hematologic function around first line of therapy. We performed logistic regression to estimate odds ratios for exclusion by strict criteria and their association with measures of patient diversity, including sex, age, race or ethnicity, and area-level socioeconomic status (SES); estimated the impact of broadening criteria on the number and distribution of eligible patients; and performed Cox regression to estimate hazard ratios for real-world overall survival (rwOS) comparing patients meeting strict versus broadened criteria. RESULTS: When applying common strict cutoffs for eligibility criteria to patients with complete data and weighting each cancer type equally, 48% of patients were eligible for clinical trials. Female (odds ratio, 1.30; 95% confidence interval [CI], 1.25 to 1.35), older (age 75+ vs. 18 to 49 years old: odds ratio, 3.04; 95% CI, 2.85 to 3.24), Latinx (odds ratio, 1.46; 95% CI, 1.39 to 1.54), non-Latinx Black (odds ratio, 1.11; 95% CI, 1.06 to 1.16), and lower-SES patients were more likely to be excluded using strict eligibility criteria. Broadening criteria increased the number of eligible patients by 78%, with the strongest impact for older, female, non-Latinx Black, and lower-SES patients. Patients who met only broadened criteria had worse rwOS versus those with strict criteria (hazard ratio, 1.31; 95% CI, 1.27 to 1.34). CONCLUSIONS: Data-driven evaluation of clinical trial eligibility criteria may optimize the eligibility of certain historically underrepresented groups and promote access to more inclusive trials. (Sponsored by Flatiron Health.).

Postprediction Inference for Clinical Characteristics Extracted With Machine Learning on Electronic Health Records.

PURPOSE: Real-world data (RWD) derived from electronic health records (EHRs) are often used to understand population-level relationships between patient characteristics and cancer outcomes. Machine learning (ML) methods enable researchers to extract characteristics from unstructured clinical notes, and represent a more cost-effective and scalable approach than manual expert abstraction. These extracted data are then used in epidemiologic or statistical models as if they were abstracted observations. Analytical results derived from extracted data in this way may differ from those given by abstracted data, and the magnitude of this difference is not directly informed by standard ML performance metrics. METHODS: In this paper, we define the task of postprediction inference, which is to recover similar estimation and inference from an ML-extracted variable that would be obtained from abstracting the variable. We consider fitting a Cox proportional hazards model that uses a binary ML-extracted variable as a covariate and evaluate four approaches for postprediction inference in this setting. The first two approaches only require the ML-predicted probability, while the latter two additionally require a labeled (human abstracted) validation data set. RESULTS: Our results for both simulated data and EHR-derived RWD from a national cohort demonstrate that we can improve inference from ML-extracted variables by leveraging a limited amount of labeled data. CONCLUSION: We describe and evaluate methods for fitting statistical models using ML-extracted variables subject to model error. We show that estimation and inference is generally valid when using extracted data from high-performing ML models. More complex methods that incorporate auxiliary labeled data provide further improvements.

A systematic approach towards missing lab data in electronic health records: A case study in non-small cell lung cancer and multiple myeloma.

Real-world data derived from electronic health records often exhibit high levels of missingness in variables, such as laboratory results, presenting a challenge for statistical analyses. We developed a systematic workflow for gathering evidence of different missingness mechanisms and performing subsequent statistical analyses. We quantify evidence for missing completely at random (MCAR) or missing at random (MAR), mechanisms using Hotelling's multivariate t-test, and random forest classifiers, respectively. We further illustrate how to apply sensitivity analyses using the not at random fully conditional specification procedure to examine changes in parameter estimates under missing not at random (MNAR) mechanisms. In simulation studies, we validated these diagnostics and compared analytic bias under different mechanisms. To demonstrate the application of this workflow, we applied it to two exemplary case studies with an advanced non-small cell lung cancer and a multiple myeloma cohort derived from a real-world oncology database. Here, we found strong evidence against MCAR, and some evidence of MAR, implying that imputation approaches that attempt to predict missing values by fitting a model to observed data may be suitable for use. Sensitivity analyses did not suggest meaningful departures of our analytic results under potential MNAR mechanisms; these results were also in line with results reported in clinical trials.

Replication of Real-World Evidence in Oncology Using Electronic Health Record Data Extracted by Machine Learning.

Meaningful real-world evidence (RWE) generation requires unstructured data found in electronic health records (EHRs) which are often missing from administrative claims; however, obtaining relevant data from unstructured EHR sources is resource-intensive. In response, researchers are using natural language processing (NLP) with machine learning (ML) techniques (i.e., ML extraction) to extract real-world data (RWD) at scale. This study assessed the quality and fitness-for-use of EHR-derived oncology data curated using NLP with ML as compared to the reference standard of expert abstraction. Using a sample of 186,313 patients with lung cancer from a nationwide EHR-derived de-identified database, we performed a series of replication analyses demonstrating some common analyses conducted in retrospective observational research with complex EHR-derived data to generate evidence. Eligible patients were selected into biomarker- and treatment-defined cohorts, first with expert-abstracted then with ML-extracted data. We utilized the biomarker- and treatment-defined cohorts to perform analyses related to biomarker-associated survival and treatment comparative effectiveness, respectively. Across all analyses, the results differed by less than 8% between the data curation methods, and similar conclusions were reached. These results highlight that high-performance ML-extracted variables trained on expert-abstracted data can achieve similar results as when using abstracted data, unlocking the ability to perform oncology research at scale.

EUS-Guided hepaticogastrostomy in a pregnant patient with Roux-en-Y hepaticojejunostomy anatomy.

Video 1EUS-Guided hepaticogastrostomy in a pregnant patient with Roux-en-Y hepaticojejunostomy anatomy.

Changes in impedance planimetry metrics predict clinical response to flexible endoscopy in Zenker's diverticulotomy.

Background and study aims Various techniques have been described for flexible endoscopic therapy for Zenker's diverticulum (ZD). Objective methods to assess myotomy effectiveness are lacking. We assessed the utility of impedance planimetry in flexible endoscopic ZD therapies and correlation with a validated symptom score. Patients and methods Patients undergoing endoscopic therapy for symptomatic ZD from February 2019 to March 2020 were included. Intraprocedural impedance planimetry was performed pre- and post-myotomy to assess esophageal diameter and distensibility index (DI). Eating Assessment Tool (EAT)-10 scores were assessed preintervention and post-intervention. Descriptive statistics were calculated. Results Thirteen patients (46 % women; mean age 80 years; 77 % peroral endoscopic myotomy technique) were included. Technical and clinical success was 100 %. No adverse events occurred. At 40 mL and 50 mL, the diameter improved (mean 2.3 mm and 2.6 mm, respectively). At 40 mL and 50 mL, the DI improved (mean 1.0 mm 2 /mmHg and 1.8 mm 2 /mmHg, respectively). EAT-10 scores improved by a mean of 15 points. Mean follow-up was 97 days. Conclusions Intraprocedural impedance planimetry may provide objective data to define success for flexible endoscopic ZD. Further research is required to corroborate these results.