Chimeric Forecasting: An experiment to leverage human judgment to improve forecasts of infectious disease using simulated surveillance data.

Forecasts of infectious agents provide public health officials advanced warning about the intensity and timing of the spread of disease. Past work has found that accuracy and calibration of forecasts is weakest when attempting to predict an epidemic peak. Forecasts from a mechanistic model would be improved if there existed accurate information about the timing and intensity of an epidemic. We presented 3000 humans with simulated surveillance data about the number of incident hospitalizations from a current and two past seasons, and asked that they predict the peak time and intensity of the underlying epidemic. We found that in comparison to two control models, a model including human judgment produced more accurate forecasts of peak time and intensity of hospitalizations during an epidemic. Chimeric models have the potential to improve our ability to predict targets of public health interest which may in turn reduce infectious disease burden.

Comparison of combination methods to create calibrated ensemble forecasts for seasonal influenza in the U.S.

The characteristics of influenza seasons vary substantially from year to year, posing challenges for public health preparation and response. Influenza forecasting is used to inform seasonal outbreak response, which can in turn potentially reduce the impact of an epidemic. The United States Centers for Disease Control and Prevention, in collaboration with external researchers, has run an annual prospective influenza forecasting exercise, known as the FluSight challenge. Uniting theoretical results from the forecasting literature with domain-specific forecasts from influenza outbreaks, we applied parametric forecast combination methods that simultaneously optimize model weights and calibrate the ensemble via a beta transformation and made adjustments to the methods to reduce their complexity. We used the beta-transformed linear pool, the finite beta mixture model, and their equal weight adaptations to produce ensemble forecasts retrospectively for the 2016/2017, 2017/2018, and 2018/2019 influenza seasons in the U.S. We compared their performance to methods that were used in the FluSight challenge to produce the FluSight Network ensemble, namely the equally weighted linear pool and the linear pool. Ensemble forecasts produced from methods with a beta transformation were shown to outperform those from the equally weighted linear pool and the linear pool for all week-ahead targets across in the test seasons based on average log scores. We observed improvements in overall accuracy despite the beta-transformed linear pool or beta mixture methods' modest under-prediction across all targets and seasons. Combination techniques that explicitly adjust for known calibration issues in linear pooling should be considered to improve probabilistic scores in outbreak settings.

Evaluation of FluSight influenza forecasting in the 2021-22 and 2022-23 seasons with a new target laboratory-confirmed influenza hospitalizations.

Accurate forecasts can enable more effective public health responses during seasonal influenza epidemics. Forecasting teams were asked to provide national and jurisdiction-specific probabilistic predictions of weekly confirmed influenza hospital admissions for one through four weeks ahead for the 2021-22 and 2022-23 influenza seasons. Across both seasons, 26 teams submitted forecasts, with the submitting teams varying between seasons. Forecast skill was evaluated using the Weighted Interval Score (WIS), relative WIS, and coverage. Six out of 23 models outperformed the baseline model across forecast weeks and locations in 2021-22 and 12 out of 18 models in 2022-23. Averaging across all forecast targets, the FluSight ensemble was the 2nd most accurate model measured by WIS in 2021-22 and the 5th most accurate in the 2022-23 season. Forecast skill and 95% coverage for the FluSight ensemble and most component models degraded over longer forecast horizons and during periods of rapid change. Current influenza forecasting efforts help inform situational awareness, but research is needed to address limitations, including decreased performance during periods of changing epidemic dynamics.

Chimeric forecasting: combining probabilistic predictions from computational models and human judgment.

Forecasts of the trajectory of an infectious agent can help guide public health decision making. A traditional approach to forecasting fits a computational model to structured data and generates a predictive distribution. However, human judgment has access to the same data as computational models plus experience, intuition, and subjective data. We propose a chimeric ensemble-a combination of computational and human judgment forecasts-as a novel approach to predicting the trajectory of an infectious agent. Each month from January, 2021 to June, 2021 we asked two generalist crowds, using the same criteria as the COVID-19 Forecast Hub, to submit a predictive distribution over incident cases and deaths at the US national level either two or three weeks into the future and combined these human judgment forecasts with forecasts from computational models submitted to the COVID-19 Forecasthub into a chimeric ensemble. We find a chimeric ensemble compared to an ensemble including only computational models improves predictions of incident cases and shows similar performance for predictions of incident deaths. A chimeric ensemble is a flexible, supportive public health tool and shows promising results for predictions of the spread of an infectious agent.

Crowdsourced Perceptions of Human Behavior to Improve Computational Forecasts of US National Incident Cases of COVID-19: Survey Study.

BACKGROUND: Past research has shown that various signals associated with human behavior (eg, social media engagement) can benefit computational forecasts of COVID-19. One behavior that has been shown to reduce the spread of infectious agents is compliance with nonpharmaceutical interventions (NPIs). However, the extent to which the public adheres to NPIs is difficult to measure and consequently difficult to incorporate into computational forecasts of infectious diseases. Soliciting judgments from many individuals (ie, crowdsourcing) can lead to surprisingly accurate estimates of both current and future targets of interest. Therefore, asking a crowd to estimate community-level compliance with NPIs may prove to be an accurate and predictive signal of an infectious disease such as COVID-19. OBJECTIVE: We aimed to show that crowdsourced perceptions of compliance with NPIs can be a fast and reliable signal that can predict the spread of an infectious agent. We showed this by measuring the correlation between crowdsourced perceptions of NPIs and US incident cases of COVID-19 1-4 weeks ahead, and evaluating whether incorporating crowdsourced perceptions improves the predictive performance of a computational forecast of incident cases. METHODS: For 36 weeks from September 2020 to April 2021, we asked 2 crowds 21 questions about their perceptions of community adherence to NPIs and public health guidelines, and collected 10,120 responses. Self-reported state residency was compared to estimates from the US census to determine the representativeness of the crowds. Crowdsourced NPI signals were mapped to 21 mean perceived adherence (MEPA) signals and analyzed descriptively to investigate features, such as how MEPA signals changed over time and whether MEPA time series could be clustered into groups based on response patterns. We investigated whether MEPA signals were associated with incident cases of COVID-19 1-4 weeks ahead by (1) estimating correlations between MEPA and incident cases, and (2) including MEPA into computational forecasts. RESULTS: The crowds were mostly geographically representative of the US population with slight overrepresentation in the Northeast. MEPA signals tended to converge toward moderate levels of compliance throughout the survey period, and an unsupervised analysis revealed signals clustered into 4 groups roughly based on the type of question being asked. Several MEPA signals linearly correlated with incident cases of COVID-19 1-4 weeks ahead at the US national level. Including questions related to social distancing, testing, and limiting large gatherings increased out-of-sample predictive performance for probabilistic forecasts of incident cases of COVID-19 1-3 weeks ahead when compared to a model that was trained on only past incident cases. CONCLUSIONS: Crowdsourced perceptions of nonpharmaceutical adherence may be an important signal to improve forecasts of the trajectory of an infectious agent and increase public health situational awareness.

An expert judgment model to predict early stages of the COVID-19 pandemic in the United States.

From February to May 2020, experts in the modeling of infectious disease provided quantitative predictions and estimates of trends in the emerging COVID-19 pandemic in a series of 13 surveys. Data on existing transmission patterns were sparse when the pandemic began, but experts synthesized information available to them to provide quantitative, judgment-based assessments of the current and future state of the pandemic. We aggregated expert predictions into a single "linear pool" by taking an equally weighted average of their probabilistic statements. At a time when few computational models made public estimates or predictions about the pandemic, expert judgment provided (a) falsifiable predictions of short- and long-term pandemic outcomes related to reported COVID-19 cases, hospitalizations, and deaths, (b) estimates of latent viral transmission, and (c) counterfactual assessments of pandemic trajectories under different scenarios. The linear pool approach of aggregating expert predictions provided more consistently accurate predictions than any individual expert, although the predictive accuracy of a linear pool rarely provided the most accurate prediction. This work highlights the importance that an expert linear pool could play in flexibly assessing a wide array of risks early in future emerging outbreaks, especially in settings where available data cannot yet support data-driven computational modeling.

Aggregating Human Judgment Probabilistic Predictions of Coronavirus Disease 2019 Transmission, Burden, and Preventive Measures.

Aggregated human judgment forecasts for coronavirus disease 2019 (COVID-19) targets of public health importance are accurate, often outperforming computational models. Our work shows that aggregated human judgment forecasts for infectious agents are timely, accurate, and adaptable, and can be used as a tool to aid public health decision making during outbreaks.

Aggregating human judgment probabilistic predictions of COVID-19 transmission, burden, and preventative measures.

Aggregated human judgment forecasts for COVID-19 targets of public health importance are accurate, often outperforming computational models. Our work shows aggregated human judgment forecasts for infectious agents are timely, accurate, and adaptable, and can be used as tool to aid public health decision making during outbreaks.

Aggregating human judgment probabilistic predictions of the safety, efficacy, and timing of a COVID-19 vaccine.

Safe, efficacious vaccines were developed to reduce the transmission of SARS-CoV-2 during the COVID-19 pandemic. But in the middle of 2020, vaccine effectiveness, safety, and the timeline for when a vaccine would be approved and distributed to the public was uncertain. To support public health decision making, we solicited trained forecasters and experts in vaccinology and infectious disease to provide monthly probabilistic predictions from July to September of 2020 of the efficacy, safety, timing, and delivery of a COVID-19 vaccine. We found, that despite sparse historical data, a linear pool-a combination of human judgment probabilistic predictions-can quantify the uncertainty in clinical significance and timing of a potential vaccine. The linear pool underestimated how fast a therapy would show a survival benefit and the high efficacy of approved COVID-19 vaccines. However, the linear pool did make an accurate prediction for when a vaccine would be approved by the FDA. Compared to individual forecasters, the linear pool was consistently above the median of the most accurate forecasts. A linear pool is a fast and versatile method to build probabilistic predictions of a developing vaccine that is robust to poor individual predictions. Though experts and trained forecasters did underestimate the speed of development and the high efficacy of a SARS-CoV-2 vaccine, linear pool predictions can improve situational awareness for public health officials and for the public make clearer the risks, rewards, and timing of a vaccine.

Adaptively stacking ensembles for influenza forecasting.

Seasonal influenza infects between 10 and 50 million people in the United States every year. Accurate forecasts of influenza and influenza-like illness (ILI) have been named by the CDC as an important tool to fight the damaging effects of these epidemics. Multi-model ensembles make accurate forecasts of seasonal influenza, but current operational ensemble forecasts are static: they require an abundance of past ILI data and assign fixed weights to component models at the beginning of a season, but do not update weights as new data on component model performance is collected. We propose an adaptive ensemble that (i) does not initially need data to combine forecasts and (ii) finds optimal weights which are updated week-by-week throughout the influenza season. We take a regularized likelihood approach and investigate this regularizer's ability to impact adaptive ensemble performance. After finding an optimal regularization value, we compare our adaptive ensemble to an equal-weighted and static ensemble. Applied to forecasts of short-term ILI incidence at the regional and national level, our adaptive model outperforms an equal-weighted ensemble and has similar performance to the static ensemble using only a fraction of the data available to the static ensemble. Needing no data at the beginning of an epidemic, an adaptive ensemble can quickly train and forecast an outbreak, providing a practical tool to public health officials looking for a forecast to conform to unique features of a specific season.

Aggregating predictions from experts: a review of statistical methods, experiments, and applications.

Forecasts support decision making in a variety of applications. Statistical models can produce accurate forecasts given abundant training data, but when data is sparse or rapidly changing, statistical models may not be able to make accurate predictions. Expert judgmental forecasts-models that combine expert-generated predictions into a single forecast-can make predictions when training data is limited by relying on human intuition. Researchers have proposed a wide array of algorithms to combine expert predictions into a single forecast, but there is no consensus on an optimal aggregation model. This review surveyed recent literature on aggregating expert-elicited predictions. We gathered common terminology, aggregation methods, and forecasting performance metrics, and offer guidance to strengthen future work that is growing at an accelerated pace.

Impact of Coronary Artery Tortuosity on Outcomes Following Stenting: A Pooled Analysis From 6 Trials.

OBJECTIVES: The authors sought to determine whether coronary artery tortuosity negatively affects clinical outcomes after stent implantation. BACKGROUND: Coronary artery tortuosity is a common angiographic finding and has been associated with increased rates of early and late major adverse events after balloon angioplasty. METHODS: Individual patient data from 6 prospective, randomized stent trials were pooled. Outcomes at 30 days and 5 years following percutaneous coronary intervention of a single coronary lesion were analyzed according to the presence or absence of moderate/severe vessel tortuosity, as determined by an angiographic core laboratory. The primary endpoint was target vessel failure (TVF) (composite of cardiac death, target vessel-related myocardial infarction [TV-MI], or ischemia-driven target vessel revascularization [ID-TVR]). RESULTS: A total of 6,951 patients were included, 729 of whom (10.5%) underwent percutaneous coronary intervention in vessels with moderate/severe tortuosity. At 30 days, TVF was more frequent in patients with versus without moderate/severe tortuosity (3.8% vs. 2.4%; hazard ratio [HR]: 1.64; 95% confidence interval [CI]: 1.09 to 2.46; p = 0.02), a difference driven by a higher rate of TV-MI. At 5 years, TVF remained increased in patients with moderate/severe tortuosity (p = 0.003), driven by higher rates of TV-MI (p = 0.003) and ID-TVR (p = 0.01). Definite stent thrombosis was also greater in patients with versus without moderate/severe tortuosity (1.9% vs. 1.0%; HR: 1.86; 95% CI: 1.02 to 3.39; p = 0.04). After adjustment for baseline covariates, moderate/severe vessel tortuosity was independently associated with TV-MI and ID-TVR at 5 years (p = 0.04 for both). CONCLUSIONS: Stent implantation in vessels with moderate/severe coronary artery tortuosity is associated with increased rates of TVF due to greater rates of TV-MI and ID-TVR.

Relationship between insulin resistance, coronary plaque, and clinical outcomes in patients with acute coronary syndromes: an analysis from the PROSPECT study.

BACKGROUND: We investigated the association of insulin resistance (IR) with coronary plaque morphology and the risk of cardiovascular events in patients enrolled in the Providing Regional Observations to Study Predictors of Events in Coronary Tree (PROSPECT) study. METHODS: Patients with acute coronary syndromes (ACS) were divided based on DM status. Non-DM patients were further stratified according to homeostasis-model-assessment IR (HOMA-IR) index as insulin sensitive (IS; HOMA-IR ≤ 2), likely-IR (LIR; 2 < HOMA-IR < 5), or diabetic-IR (DIR; HOMA-IR ≥ 5). Coronary plaque characteristics were investigated by intravascular ultrasound. The primary endpoint was major adverse cardiac events (MACE); a composite of cardiac death, cardiac arrest, myocardial infarction, and rehospitalization for unstable/progressive angina. RESULTS: Among non-diabetic patients, 109 patients (21.5%) were categorized as LIR, and 65 patients (12.8%) as DIR. Patients with DIR or DM had significantly higher rates of echolucent plaque compared with LIR and IS. In addition, DIR and DM were independently associated with increased risk of MACE compared with IS (adjusted hazard ratio [aHR] 2.29, 95% confidence interval [CI] 1.22-4.29, p = 0.01 and aHR 2.12, 95% CI 1.19-3.75, p = 0.009, respectively). CONCLUSIONS: IR is common among patients with ACS. DM and advanced but not early stages of IR are independently associated with increased risk of adverse cardiovascular events. Trial Registration ClinicalTrials.gov Identifier: NCT00180466.

Impact of lesion preparation strategies on outcomes of left main PCI: The EXCEL trial.

OBJECTIVES: We examined outcomes according to lesion preparation strategy (LPS) in patients with left main coronary artery (LMCA) percutaneous coronary intervention (PCI) in the EXCEL trial. BACKGROUND: The optimal LPS for LMCA PCI is unclear. METHODS: We categorized LPS hierarchically (high to low) as: (a) rotational atherectomy (RA); (b) cutting or scoring balloon (CSB); (c) balloon angioplasty (BAL); and d) direct stenting (DIR). The primary endpoint was 3-year MACE; all-cause death, stroke, or myocardial infarction. RESULTS: Among 938 patients undergoing LMCA PCI, RA was performed in 6.0%, CSB 9.5%, BAL 71.3%, and DIR 13.2%. In patients treated with DIR, BAL, CSB, and RA, respectively, there was a progressive increase in SYNTAX score, LMCA complex bifurcation, trifurcation or calcification, number of stents, and total stent length. Any procedural complication occurred in 10.4% of cases overall, with the lowest rate in the DIR (7.4%) and highest in the RA group (16.1%) (ptrend  = .22). There were no significant differences in the 3-year rates of MACE (from RA to DIR: 17.9%, 20.2%, 14.5%, 14.7%; p = .50) or ischemia-driven revascularization (from RA to DIR: 16.8%, 10.8%, 12.3%, 14.2%; p = .65). The adjusted 3-year rates of MACE did not differ according to LPS. CONCLUSIONS: The comparable 3-year outcomes suggest that appropriate lesion preparation may be able to overcome the increased risks of complex LMCA lesion morphology.

Diastolic Function and Clinical Outcomes After Transcatheter Aortic Valve Replacement: PARTNER 2 SAPIEN 3 Registry.

BACKGROUND: Few studies have evaluated if diastolic function could predict outcomes in patients with aortic stenosis. OBJECTIVES: The authors aimed to assess the association between diastolic dysfunction (DD) and outcomes in patients with aortic stenosis undergoing transcatheter aortic valve replacement (TAVR). METHODS: Baseline, 30-day, and 1- and 2-year transthoracic echocardiograms from the PARTNER (Placement of Aortic Transcatheter Valves) 2 SAPIEN 3 registry were analyzed by a consortium of core laboratories and divided into the American Society of Echocardiography DD groups. RESULTS: Among the 1,750 included, 682 (54.4%) had grade 1 DD, 352 (28.1%) had grade 2 DD, 168 (13.4%) had grade 3 DD, and 51 (4.1%) had indeterminate DD grade. Incremental baseline grades of DD were associated with an increase in combined 1- and 2-year cardiovascular (CV) death/rehospitalization (all p < 0.002) and all-cause death at 2 years (p = 0.01) but not at 1 year. Improvement in DD grade/grade 1 DD at 30 days post-TAVR was seen in 70.8% patients. Patients with improvement in ≥1 grade of DD/grade 1 DD had reduced 1-year CV death/rehospitalization (p < 0.001) and increased 2-year survival (p = 0.01). Baseline grade 3 DD was a predictor of 1-year CV death/rehospitalization (hazard ratio: 2.73; 95% confidence interval: 1.07 to 6.98; p = 0.04). Improvement in DD grade/grade 1 DD at 30 days was protective for 1-year CV death/rehospitalizations (hazard ratio: 0.39; 95% confidence interval: 0.19 to 0.83; p = 0.01). CONCLUSIONS: In the PARTNER 2 SAPIEN 3 registry, baseline DD was a predictor of up to 2 years clinical outcomes in patients who underwent TAVR. Improvement in DD grade at 30 days was associated with improvement in short-term clinical outcomes. (The PARTNER II Trial: Placement of AoRTic TraNscathetER Valves II - PARTNER II - PARTNERII - S3 Intermediate [PARTNERII S3i]; NCT03222128; PARTNER II Trial: Placement of AoRTic TraNscathetER Valves II - High Risk and Nested Registry 7 [PII S3HR/NR7]; NCT03222141).

The win ratio approach for composite endpoints: practical guidance based on previous experience.

The win ratio was introduced in 2012 as a new method for examining composite endpoints and has since been widely adopted in cardiovascular (CV) trials. Improving upon conventional methods for analysing composite endpoints, the win ratio accounts for relative priorities of the components and allows the components to be different types of outcomes. For example, the win ratio can combine the time to death with the number of occurrences of a non-fatal outcome such as CV-related hospitalizations (CVHs) in a single hierarchical composite endpoint. The win ratio can provide greater statistical power to detect and quantify a treatment difference by using all available information contained in the component outcomes. The win ratio can also incorporate quantitative outcomes such as exercise tests or quality-of-life scores. There is a need for more practical guidance on how best to design trials using the win ratio approach. This manuscript provides an overview of the principles behind the win ratio and provides insights into how to implement the win ratio in CV trial design and reporting, including how to determine trial size.

An expert judgment model to predict early stages of the COVID-19 outbreak in the United States.

During early stages of the COVID-19 pandemic, forecasts provided actionable information about disease transmission to public health decision-makers. Between February and May 2020, experts in infectious disease modeling made weekly predictions about the impact of the pandemic in the U.S. We aggregated these predictions into consensus predictions. In March and April 2020, experts predicted that the number of COVID-19 related deaths in the U.S. by the end of 2020 would be in the range of 150,000 to 250,000, with scenarios of near 1m deaths considered plausible. The wide range of possible future outcomes underscored the uncertainty surrounding the outbreak's trajectory. Experts' predictions of measurable short-term outcomes had varying levels of accuracy over the surveys but showed appropriate levels of uncertainty when aggregated. An expert consensus model can provide important insight early on in an emerging global catastrophe.

Effect of Pressure-controlled intermittent Coronary Sinus Occlusion (PiCSO) on infarct size in anterior STEMI: PiCSO in ACS study.

BACKGROUND: The aim of this clinical research was to investigate the effects of Pressure-controlled intermittent Coronary Sinus Occlusion (PiCSO) on infarct size at 5 days after primary percutaneous coronary intervention (pPCI) in patients with ST-segment elevation myocardial infarction (STEMI). METHODS AND RESULTS: This comparative study was carried out in four UK hospitals. Forty-five patients with anterior STEMI presenting within 12 h of symptom onset received pPCI plus PiCSO (initiated after reperfusion; n = 45) and were compared with a propensity score-matched control cohort from INFUSE-AMI (n = 80). Infarct size (% of LV mass, median [interquartile range]) measured by cardiac magnetic resonance (CMR) at day 5 was significantly lower in the PiCSO group (14.3% [95% CI 9.2-19.4%] vs. 21.2% [95% CI 18.0-24.4%]; p = 0.023). There were no major adverse cardiac events (MACE) related to the PiCSO intervention. CONCLUSIONS: PiCSO, as an adjunct to pPCI, was associated with a lower infarct size at 5 days after anterior STEMI in a propensity score-matched population.

Global Approach to High Bleeding Risk Patients With Polymer-Free Drug-Coated Coronary Stents: The LF II Study.

BACKGROUND: High bleeding risk (HBR) patients undergoing percutaneous coronary intervention have been widely excluded from randomized device registration trials. The LF study (LEADERS FREE) reported superior outcomes of HBR patients receiving 30-day dual antiplatelet therapy after percutaneous coronary intervention with a polymer-free drug-coated stent (DCS). LFII was designed to assess the reproducibility and generalizability of the benefits of DCS observed in LF to inform the US Food and Drug Administration in a device registration decision. METHODS: LFII was a single-arm study using HBR inclusion/exclusion criteria and 30-day dual antiplatelet therapy after percutaneous coronary intervention with DCS, identical to LF. The 365-day rates of the primary effectiveness (clinically indicated target lesion revascularization) and safety (composite cardiac death and myocardial infarction) end points were reported using a propensity-stratified analysis compared with the LF bare metal stent arm patients as controls. RESULTS: A total of 1203 LFII patients were enrolled with an average 1.7 HBR criteria per patient, including 60.7% >75 years of age, 34.1% on anticoagulants, and 14.7% with renal failure. Propensity-adjusted 365-day clinically indicated target lesion revascularization was significantly lower with DCS (7.2% versus 9.2%; hazard ratio, 0.72 [95% CI, 0.52-0.98]; P=0.0338 for superiority), as was the primary safety (cardiac death and myocardial infarction) composite (9.3% versus 12.4%; hazard ratio, 0.72 [95% CI, 0.55-0.94]; P=0.0150 for superiority). Stent thrombosis rates were 2.0% DCS and 2.2% bare metal stent. Major bleeding at 1 year occurred in 7.2% DCS patients and 7.2% bare metal stent. CONCLUSIONS: LFII reproduces the results of the DCS arm of LF in an independent, predominantly North American cohort of HBR patients.