Demystifying Statistical Inference When Using Machine Learning in Causal Research.

In this issue, Naimi et al. (Am J Epidemiol. XXXX;XXX(XX):XXXX-XXXX) discuss a critical topic in public health and beyond: obtaining valid statistical inference when using machine learning in causal research. In doing so, the authors review recent prominent methodological work and recommend: (i) double robust estimators, such as targeted maximum likelihood estimation (TMLE); (ii) ensemble methods, such as Super Learner, to combine predictions from a diverse library of algorithms, and (iii) sample-splitting to reduce bias and improve inference. We largely agree with these recommendations. In this commentary, we highlight the critical importance of the Super Learner library. Specifically, in both simulation settings considered by the authors, we demonstrate that low bias and valid statistical inference can be achieved using TMLE without sample-splitting and with a Super Learner library that excludes tree-based methods but includes regression splines. Whether extremely data-adaptive algorithms and sample-splitting are needed depends on the specific problem and should be informed by simulations reflecting the specific application. More research is needed on practical recommendations for selecting among these options in common situations arising in epidemiology.

Prospective Evaluation of Galactomannan and (1→3) ß-d-Glucan Assays as Diagnostic Tools for Invasive Fungal Disease in Children, Adolescents, and Young Adults With Acute Myeloid Leukemia Receiving Fungal Prophylaxis.

BACKGROUND: Patients receiving chemotherapy for acute myeloid leukemia (AML) are at high risk for invasive fungal disease (IFD). Diagnosis of IFD is challenging, leading to interest in fungal biomarkers. The objective was to define the utility of surveillance testing with Platelia Aspergillus galactomannan (GM) enzyme immunoassay (EIA) and Fungitell ß-d-glucan (BDG) assay in children with AML receiving antifungal prophylaxis. METHODS: Twice-weekly surveillance blood testing with GM EIA and BDG assay was performed during periods of neutropenia in the context of a randomized trial of children, adolescents, and young adults with AML allocated to fluconazole or caspofungin prophylaxis. Proven or probable IFD was adjudicated using blinded central reviewers. The sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for Platelia and Fungitell assays alone and in combination for the outcomes of proven and probable invasive aspergillosis (IA) or invasive candidiasis (IC). RESULTS: Among 471 patients enrolled, 425 participants (209 fluconazole and 216 caspofungin) contributed ≥1 blood specimen. In total, 6103 specimens were evaluated, with a median of 15 specimens per patient (range 1-43). The NPV was >99% for GM EIA and BDG assay alone and in combination. However, there were no true positive results, resulting in sensitivity and PPV for each assay of 0%. CONCLUSIONS: The GM EIA and the BDG assay alone or in combination were not successful at detecting IA or IC during periods of neutropenia in children, adolescents, and young adults with AML receiving antifungal prophylaxis. Utilization of these assays for surveillance in this clinical setting should be discouraged.

Use of a metalearner to predict emergency medical services demand in an urban setting.

OBJECTIVE: To develop and internally validate a metalearner algorithm to predict the hourly rate of emergency medical services (EMS) dispatches in an urban setting. METHODS: We performed an analysis of EMS data from New York City between years 2015-2019. Our outcome was hourly EMS dispatches, expressed as continuous data. Hours were split into derivation (75%) and validation (25%) datasets. Candidate variables included averages of prior rates, temporal and weather characteristics. We used a metalearner to evaluate and aggregate individual learners (generalized linear model, generalized additive model, random forest, multivariable adaptive regression splines, and extreme gradient boost). Four models were investigated: 1) temporal variables, 2) weather and temporal variables, and datasets in which weather data were lagged by 3) six and 4) twelve hours. In exploratory analyses, we constructed learners for high acuity and trauma encounters. RESULTS: 7,364,275 EMS dispatches occurred during the 43,823-hour period. When using temporal variables, the mean absolute error (MAE) rate was 11.5 dispatches in the validation dataset. These were slightly improved following incorporation of weather variables (MAE 11.3). When using 6- and 12-hour lagged weather variables, learners demonstrated lower accuracy (MAE 11.8 in 6-hour lagged datasets; 12.2 in 12-hour lagged dataset). All models had a coefficient of determination (R2) ≥0.91. The extreme gradient boosting and random forest learners were assigned the highest coefficients. In an investigation of variable importance, hour of day and average EMS dispatches over the previous six hours were the most important variables in both the extreme gradient boosting and random forest learners. The algorithm performed well at predicting frequently occurring peaks, with greater challenges at both extremes. Learners created high-acuity and for trauma-related encounters demonstrated superior MAE, but with lower R2 in the validation cohort (MAE 6.9 and R2 0.84 for high acuity encounters; MAE 5.3 and R2 0.79 for trauma in learners using time and weather variables). CONCLUSION: We developed an ensemble machine learning algorithm to predict EMS dispatches in an urban setting. These models demonstrated high accuracy, with MAEs <12 per hour in all. These algorithms may carry benefit in the real-time prediction of EMS responses, allowing for improved resource utilization.

A UNIFIED STUDY OF NONPARAMETRIC INFERENCE FOR MONOTONE FUNCTIONS.

The problem of nonparametric inference on a monotone function has been extensively studied in many particular cases. Estimators considered have often been of so-called Grenander type, being representable as the left derivative of the greatest convex minorant or least concave majorant of an estimator of a primitive function. In this paper, we provide general conditions for consistency and pointwise convergence in distribution of a class of generalized Grenander-type estimators of a monotone function. This broad class allows the minorization or majoratization operation to be performed on a data-dependent transformation of the domain, possibly yielding benefits in practice. Additionally, we provide simpler conditions and more concrete distributional theory in the important case that the primitive estimator and data-dependent transformation function are asymptotically linear. We use our general results in the context of various well-studied problems, and show that we readily recover classical results established separately in each case. More importantly, we show that our results allow us to tackle more challenging problems involving parameters for which the use of flexible learning strategies appears necessary. In particular, we study inference on monotone density and hazard functions using informatively right-censored data, extending the classical work on independent censoring, and on a covariate-marginalized conditional mean function, extending the classical work on monotone regression functions.

Correcting an estimator of a multivariate monotone function with isotonic regression.

In many problems, a sensible estimator of a possibly multivariate monotone function may fail to be monotone. We study the correction of such an estimator obtained via projection onto the space of functions monotone over a finite grid in the domain. We demonstrate that this corrected estimator has no worse supremal estimation error than the initial estimator, and that analogously corrected confidence bands contain the true function whenever the initial bands do, at no loss to band width. Additionally, we demonstrate that the corrected estimator is asymptotically equivalent to the initial estimator if the initial estimator satisfies a stochastic equicontinuity condition and the true function is Lipschitz and strictly monotone. We provide simple sufficient conditions in the special case that the initial estimator is asymptotically linear, and illustrate the use of these results for estimation of a G-computed distribution function. Our stochastic equicontinuity condition is weaker than standard uniform stochastic equicontinuity, which has been required for alternative correction procedures. This allows us to apply our results to the bivariate correction of the local linear estimator of a conditional distribution function known to be monotone in its conditioning argument. Our experiments suggest that the projection step can yield significant practical improvements.

Causal Isotonic Regression.

In observational studies, potential confounders may distort the causal relationship between an exposure and an outcome. However, under some conditions, a causal dose-response curve can be recovered using the G-computation formula. Most classical methods for estimating such curves when the exposure is continuous rely on restrictive parametric assumptions, which carry significant risk of model misspecification. Nonparametric estimation in this context is challenging because in a nonparametric model these curves cannot be estimated at regular rates. Many available nonparametric estimators are sensitive to the selection of certain tuning parameters, and performing valid inference with such estimators can be difficult. In this work, we propose a nonparametric estimator of a causal dose-response curve known to be monotone. We show that our proposed estimation procedure generalizes the classical least-squares isotonic regression estimator of a monotone regression function. Specifically, it does not involve tuning parameters, and is invariant to strictly monotone transformations of the exposure variable. We describe theoretical properties of our proposed estimator, including its irregular limit distribution and the potential for doubly-robust inference. Furthermore, we illustrate its performance via numerical studies, and use it to assess the relationship between BMI and immune response in HIV vaccine trials.

Methods for comparing durability of immune responses between vaccine regimens in early-phase trials.

The ability to produce a long-lasting, or durable, immune response is a crucial characteristic of many highly effective vaccines. A goal of early-phase vaccine trials is often to compare the immune response durability of multiple tested vaccine regimens. One parameter for measuring immune response durability is the area under the mean post-peak log immune response profile. In this paper, we compare immune response durability across vaccine regimens within and between two phase I trials of DNA-primed HIV vaccine regimens, HVTN 094 and HVTN 096. We compare four estimators of this durability parameter and the resulting statistical inferences for comparing vaccine regimens. Two of these estimators use the trapezoid rule as an empirical approximation of the area under the marginal log response curve, and the other two estimators are based on linear and nonlinear models for the marginal mean log response. We conduct a simulation study to compare the four estimators, provide guidance on estimator selection, and use the nonlinear marginal mean model to analyze immunogenicity data from the two HIV vaccine trials.

The role of elective neck dissection in high-grade parotid malignancy: A hospital-based cohort study.

OBJECTIVES/HYPOTHESIS: The role of elective neck dissection (END) in patients with clinically N0 (cN0), high-grade parotid carcinoma is unclear. The objective of this study was to assess the association between END and survival in patients with cN0, high-grade parotid carcinoma. STUDY DESIGN: Retrospective, multicenter cohort study. METHODS: A review of hospital-based cases from the National Cancer Data Base was performed. Participants included patients diagnosed with cN0, high-grade parotid cancer between January 1, 2004 and December 31, 2013. The primary exposure was receipt of neck dissection. Secondary exposures included receipt of adjuvant radiation and/or chemotherapy. Univariate and multivariate survival analyses were performed. Unadjusted and adjusted survival estimates were determined. RESULTS: Overall, 1,547 patients were included, with a median follow-up time of 48 months. END did not have a statistically significant effect on 3-year survival (3-year: 69.9%, 95% confidence interval [CI]: 67.2 to 72.6). Survival at 3-years among those not receiving END was 66.1% (95% CI: 62.7 to 69.5). Parotidectomy and adjuvant radiotherapy had the strongest effect on mortality. There was no difference in 3-year survival among patients who underwent parotidectomy and adjuvant radiation stratified by receipt of END nor did END have a statistically significant effect on survival in mucoepidermoid carcinoma, adenocarcinoma, high-risk histology, high T stage, or academic center treatment subgroups. CONCLUSIONS: END did not have a statistically significant effect on survival among cN0 patients with high-grade parotid cancer when taking into account receipt of adjuvant therapy and confounding. The role of END on survival and locoregional control remains to be further elucidated in prospective studies. LEVEL OF EVIDENCE: 4 Laryngoscope, 130:1487-1495, 2020.

Multiresolution Network Models.

Many existing statistical and machine learning tools for social network analysis focus on a single level of analysis. Methods designed for clustering optimize a global partition of the graph, whereas projection-based approaches (e.g., the latent space model in the statistics literature) represent in rich detail the roles of individuals. Many pertinent questions in sociology and economics, however, span multiple scales of analysis. Further, many questions involve comparisons across disconnected graphs that will, inevitably be of different sizes, either due to missing data or the inherent heterogeneity in real-world networks. We propose a class of network models that represent network structure on multiple scales and facilitate comparison across graphs with different numbers of individuals. These models differentially invest modeling effort within subgraphs of high density, often termed communities, while maintaining a parsimonious structure between said subgraphs. We show that our model class is projective, highlighting an ongoing discussion in the social network modeling literature on the dependence of inference paradigms on the size of the observed graph. We illustrate the utility of our method using data on household relations from Karnataka, India. Supplementary material for this article is available online.

Effect of Dengue Serostatus on Dengue Vaccine Safety and Efficacy.

BACKGROUND: In efficacy trials of a tetravalent dengue vaccine (CYD-TDV), excess hospitalizations for dengue were observed among vaccine recipients 2 to 5 years of age. Precise risk estimates according to observed dengue serostatus could not be ascertained because of the limited numbers of samples collected at baseline. We developed a dengue anti-nonstructural protein 1 (NS1) IgG enzyme-linked immunosorbent assay and used samples from month 13 to infer serostatus for a post hoc analysis of safety and efficacy. METHODS: In a case-cohort study, we reanalyzed data from three efficacy trials. For the principal analyses, we used baseline serostatus determined on the basis of measured (when baseline values were available) or imputed (when baseline values were missing) titers from a 50% plaque-reduction neutralization test (PRNT50), with imputation conducted with the use of covariates that included the month 13 anti-NS1 assay results. The risk of hospitalization for virologically confirmed dengue (VCD), of severe VCD, and of symptomatic VCD according to dengue serostatus was estimated by weighted Cox regression and targeted minimum loss-based estimation. RESULTS: Among dengue-seronegative participants 2 to 16 years of age, the cumulative 5-year incidence of hospitalization for VCD was 3.06% among vaccine recipients and 1.87% among controls, with a hazard ratio (vaccine vs. control) through data cutoff of 1.75 (95% confidence interval [CI], 1.14 to 2.70). Among dengue-seronegative participants 9 to 16 years of age, the cumulative incidence of hospitalization for VCD was 1.57% among vaccine recipients and 1.09% among controls, with a hazard ratio of 1.41 (95% CI, 0.74 to 2.68). Similar trends toward a higher risk among seronegative vaccine recipients than among seronegative controls were also found for severe VCD. Among dengue-seropositive participants 2 to 16 years of age and those 9 to 16 years of age, the cumulative incidence of hospitalization for VCD was 0.75% and 0.38%, respectively, among vaccine recipients and 2.47% and 1.88% among controls, with hazard ratios of 0.32 (95% CI, 0.23 to 0.45) and 0.21 (95% CI, 0.14 to 0.31). The risk of severe VCD was also lower among seropositive vaccine recipients than among seropositive controls. CONCLUSIONS: CYD-TDV protected against severe VCD and hospitalization for VCD for 5 years in persons who had exposure to dengue before vaccination, and there was evidence of a higher risk of these outcomes in vaccinated persons who had not been exposed to dengue. (Funded by Sanofi Pasteur; ClinicalTrials.gov numbers, NCT00842530 , NCT01983553 , NCT01373281 , and NCT01374516 .).