Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States.

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks.

Predicting Response to Intravesical BCG in High-Risk NMIBC Using an Artificial Intelligence-Powered Pathology Assay: Development and Validation in an International 12-Center Cohort.

PURPOSE: There are few markers to identify those likely to recur or progress after treatment with intravesical bacillus Calmette-Guérin (BCG). We developed and validated artificial intelligence-based histologic assays that extract interpretable features from transurethral resection of bladder tumor digitized pathology images to predict risk of recurrence, progression, development of BCG-unresponsive disease, and cystectomy. MATERIALS AND METHODS: Pre-BCG resection-derived whole-slide images and clinical data were obtained for high-risk NMIBC cases treated with BCG from 12 centers and were analyzed through a segmentation and feature extraction pipeline. Features associated with clinical outcomes were defined and tested on independent development and validation cohorts. Cases were classified into high or low risk for recurrence, progression, BCG-unresponsive disease, and cystectomy. RESULTS: Nine hundred forty-four cases (development: 303, validation: 641, median follow-up: 36 months) representative of the intended use population were included (high-grade Ta: 34.1%, high-grade T1: 54.8%; carcinoma in situ only: 11.1%, any carcinoma in situ: 31.4%). In the validation cohort, "high recurrence risk" cases had inferior high-grade recurrence-free survival vs "low recurrence risk" cases (HR, 2.08, P < .0001). "High progression risk" patients had poorer progression-free survival (HR, 3.87, P < .001) and higher risk of cystectomy (HR, 3.35, P < .001) than "low progression risk" patients. Cases harboring the BCG-unresponsive disease signature had a shorter time to development of BCG-unresponsive disease than cases without the signature (HR, 2.31, P < .0001). AI assays provided predictive information beyond clinicopathologic factors. CONCLUSIONS: We developed and validated AI-based histologic assays that identify high-risk NMIBC cases at higher risk of recurrence, progression, BCG-unresponsive disease, and cystectomy, potentially aiding clinical decision making.