A retrospective assessment of COVID-19 model performance in the USA.

Coronavirus disease 2019 (COVID-19) forecasts from over 100 models are readily available. However, little published information exists regarding the performance of their uncertainty estimates (i.e. probabilistic performance). To evaluate their probabilistic performance, we employ the classical model (CM), an established method typically used to validate expert opinion. In this analysis, we assess both the predictive and probabilistic performance of COVID-19 forecasting models during 2021. We also compare the performance of aggregated forecasts (i.e. ensembles) based on equal and CM performance-based weights to an established ensemble from the Centers for Disease Control and Prevention (CDC). Our analysis of forecasts of COVID-19 mortality from 22 individual models and three ensembles across 49 states indicates that-(i) good predictive performance does not imply good probabilistic performance, and vice versa; (ii) models often provide tight but inaccurate uncertainty estimates; (iii) most models perform worse than a naive baseline model; (iv) both the CDC and CM performance-weighted ensembles perform well; but (v) while the CDC ensemble was more informative, the CM ensemble was more statistically accurate across states. This study presents a worthwhile method for appropriately assessing the performance of probabilistic forecasts and can potentially improve both public health decision-making and COVID-19 modelling.

Health benefits of decreases in on-road transportation emissions in the United States from 2008 to 2017.

Decades of air pollution regulation have yielded enormous benefits in the United States, but vehicle emissions remain a climate and public health issue. Studies have quantified the vehicle-related fine particulate matter (PM2.5)-attributable mortality but lack the combination of proper counterfactual scenarios, latest epidemiological evidence, and detailed spatial resolution; all needed to assess the benefits of recent emission reductions. We use this combination to assess PM2.5-attributable health benefits and also assess the climate benefits of on-road emission reductions between 2008 and 2017. We estimate total benefits of $270 (190 to 480) billion in 2017. Vehicle-related PM2.5-attributable deaths decreased from 27,700 in 2008 to 19,800 in 2017; however, had per-mile emission factors remained at 2008 levels, 48,200 deaths would have occurred in 2017. The 74% increase from 27,700 to 48,200 PM2.5-attributable deaths with the same emission factors is due to lower baseline PM2.5 concentrations (+26%), more vehicle miles and fleet composition changes (+22%), higher baseline mortality (+13%), and interactions among these (+12%). Climate benefits were small (3 to 19% of the total). The percent reductions in emissions and PM2.5-attributable deaths were similar despite an opportunity to achieve disproportionately large health benefits by reducing high-impact emissions of passenger light-duty vehicles in urban areas. Increasingly large vehicles and an aging population, increasing mortality, suggest large health benefits in urban areas require more stringent policies. Local policies can be effective because high-impact primary PM2.5 and NH3 emissions disperse little outside metropolitan areas. Complementary national-level policies for NOx are merited because of its substantial impacts-with little spatial variability-and dispersion across states and metropolitan areas.

Long-term effect of exposure to lower concentrations of air pollution on mortality among US Medicare participants and vulnerable subgroups: a doubly-robust approach.

BACKGROUND: Long-term exposure to air pollution has been linked with an increase in risk of mortality. Whether existing US Environmental Protection Agency standards are sufficient to protect health is unclear. Our study aimed to examine the relationship between exposure to lower concentrations of air pollution and the risk of mortality. METHODS: Our nationwide cohort study investigated the effect of annual average exposure to air pollutants on all-cause mortality among Medicare enrolees from the beginning of 2000 to the end of 2016. Patients entered the cohort in the month of January following enrolment and were followed up until the end of the study period in 2016 or death. We restricted our analyses to participants who had only been exposed to lower concentrations of pollutants over the study period, specifically particulate matter less than 2·5 µg/m3 in diameter (PM2·5) at a concentration of up to 12 µg/m3, nitrogen dioxide (NO2) at a concentration of up to 53 parts per billion (ppb), and summer ozone (O3) at concentrations of up to 50 ppb. We adjusted for two types of covariates, which were individual level and postal code-level variables. We used a doubly-robust additive model to estimate the change in risk. We further looked at effect-measure modification by stratification on the basis of demographic and socioeconomic characteristics. FINDINGS: We found an increased risk of mortality with all three pollutants. Each 1 µg/m3 increase in annual PM2·5 concentrations increased the absolute annual risk of death by 0·073% (95% CI 0·071-0·076). Each 1 ppb increase in annual NO2 concentrations increased the annual risk of death by 0·003% (0·003-0·004), and each 1 ppb increase in summer O3 concentrations increased the annual risk of death by 0·081% (0·080-0·083). This increase translated to approximately 11 540 attributable deaths (95% CI 11 087-11 992) for PM2·5, 1176 attributable deaths (998-1353) for NO2, and 15 115 attributable deaths (14 896-15 333) for O3 per year for each unit increase in pollution concentrations. The effects were higher in certain subgroups, including individuals living in areas of low socioeconomic status. Long-term exposure to permissible concentrations of air pollutants increases the risk of mortality. FUNDING: The US Environmental Protection Agency, National Institute of Environmental Health Services, and Health Effects Institute.

Long-Term Association of Air Pollution and Hospital Admissions Among Medicare Participants Using a Doubly Robust Additive Model.

BACKGROUND: Studies examining the nonfatal health outcomes of exposure to air pollution have been limited by the number of pollutants studied and focus on short-term exposures. METHODS: We examined the relationship between long-term exposure to fine particulate matter with an aerodynamic diameter <2.5 micrometers (PM2.5), NO2, and tropospheric ozone and hospital admissions for 4 cardiovascular and respiratory outcomes (myocardial infarction, ischemic stroke, atrial fibrillation and flutter, and pneumonia) among the Medicare population of the United States. We used a doubly robust method for our statistical analysis, which relies on both inverse probability weighting and adjustment in the outcome model to account for confounding. The results from this regression are on an additive scale. We further looked at this relationship at lower pollutant concentrations, which are consistent with typical exposure levels in the United States, and among potentially susceptible subgroups. RESULTS: Long-term exposure to fine PM2.5 was associated with an increased risk of all outcomes with the highest effect seen for stroke with a 0.0091% (95% CI, 0.0086-0.0097) increase in the risk of stroke for each 1-µg/m3 increase in annual levels. This translated to 2536 (95% CI, 2383-2691) cases of hospital admissions with ischemic stroke per year, which can be attributed to each 1-unit increase in fine particulate matter levels among the study population. NO2 was associated with an increase in the risk of admission with stroke by 0.00059% (95% CI, 0.00039-0.00075) and atrial fibrillation by 0.00129% (95% CI, 0.00114-0.00148) per ppb and tropospheric ozone was associated with an increase in the risk of admission with pneumonia by 0.00413% (95% CI, 0.00376-0.00447) per parts per billion. At lower concentrations, all pollutants were consistently associated with an increased risk for all our studied outcomes. CONCLUSIONS: Long-term exposure to air pollutants poses a significant risk to cardiovascular and respiratory health among the elderly population in the United States, with the greatest increase in the association per unit of exposure occurring at lower concentrations.

Mortality Benefits and Control Costs of Improving Air Quality in Mexico City: The Case of Heavy Duty Diesel Vehicles.

Diesel vehicles are significant contributors to air pollution in Mexico City. We estimate the costs and mortality benefits of retrofitting heavy-duty vehicles with particulate filters and oxidation catalysts. The feasibility and cost-effectiveness of controls differ by vehicle model-year and type. We evaluate 1985 to 2014 model-year vehicles from 10 vehicle classes and five model-year groups. Our analysis shows that retrofitting all vehicles with the control that maximizes expected net benefits for that vehicle type and model-year group has the potential to reduce emissions of primary fine particles (PM2.5 ) by 950 metric tons/year; cut the population-weighted annual mean concentration of PM2.5 in Mexico City by 0.90 µg/m3 ; reduce the annual number of deaths attributable to air pollution by over 80; and generate expected annual health benefits of close to 250 million US$. These benefits outweigh expected costs of 92 million US$ per year. Diesel retrofits are but one step that should viewed in the context of other efforts--such as development of an integrated public transportation system, promotion of the rational use of cars, reduction of emissions from industrial sources and fires, and redesign of the Mexico City Metropolitan Area to reduce urban sprawl--that must be analyzed and implemented to substantially control air pollution and protect public health. Even if considering other potential public health interventions, which would offer greater benefits at the same or lower costs, only by conducting, promoting, and publishing this sort of analyses, we can make strides to improve public health cost-effectively.

Cost-effectiveness analysis of three algorithms for diagnosing primary ciliary dyskinesia: a simulation study.

BACKGROUND: Primary Ciliary Dyskinesia (PCD) diagnosis relies on a combination of tests which may include (a) nasal Nitric Oxide (nNO), (b) High Speed Video Microscopy (HSVM) and (c) Transmission Electron Microscopy (TEM). There is variability in the availability of these tests and lack of universal agreement whether diagnostic tests should be performed in sequence or in parallel. We assessed three combinations of tests for PCD diagnosis and estimated net sensitivity and specificity as well as cost-effectiveness (CE) and incremental cost-effectiveness (ICE) ratios. METHODS AND RESULTS: A hypothetical initial population of 1000 referrals (expected 320 PCD patients) was followed through a probabilistic decision analysis model which was created to assess the CE of three diagnostic algorithms (a) nNO + TEM in sequence, (b) nNO + HSVM in sequence and (c) nNO/HSVM in parallel followed, in cases with conflicting results, by confirmatory TEM (nNO/HSVM+TEM). Number of PCD patients identified, CE and ICE ratios were calculated using Monte Carlo simulations. Out of 320 expected PCD patients, 313 were identified by nNO/HSVM+TEM, 274 with nNO + HSVM and 198 with nNO + TEM. The nNO/HSVM+TEM had the highest mean annual cost (€209 K) followed by nNO + TEM (€150 K) and nNO + HSVM (€136 K). The nNO + HSVM algorithm dominated the nNO + TEM algorithm (less costly and more effective). The ICE ratio for nNO/HSVM+TEM was €2.1 K per additional PCD patient identified. CONCLUSIONS: The diagnostic algorithm (nNO/HSVM+TEM) with parallel testing outperforms algorithms with tests in sequence. These findings, can inform the dialogue on the development of evidence-based guidelines for PCD diagnostic testing. Future research in understudied aspects of the disease, such as PCD-related quality of life and PCD-associated costs, is needed to help the better implementation of these guidelines across various healthcare systems.

A probabilistic characterization of the health benefits of reducing methyl mercury intake in the United States.

We developed a probabilistic model to characterize the plausible distribution of health and economic benefits that would accrue to the U.S. population following reduction of methyl mercury (MeHg) exposure. MeHg, a known human developmental neurotoxicant, may increase fatal heart attack risks. Model parameters reflect current understanding of the relationships between MeHg intake, health risks, and societal valuation of these risks. The expected monetary value of the annual health benefits generated by a 10% reduction in U.S. population exposure to MeHg for one year is $860 million; 80% of this is associated with reductions in fatal heart attacks and the remainder with IQ gains. The plausible distribution of the benefits is quite broad with 5th and 95th percentile estimates of approximately $50 million and $3.5 billion, respectively. The largest source of uncertainty is whether epidemiological associations between MeHg exposure and fatal heart attacks reflect causality. The next largest sources of uncertainty concern the slope of the relationship between maternal MeHg exposure and reduced intelligence among children and whether this relationship exhibits a threshold. Our analysis suggests that the possible causal relationship between MeHg exposure and fatal heart attacks should be better characterized, using additional epidemiological studies and formally elicited expert judgment.

The hydrogen-bonding transition and isotope-dependent negative thermal expansion in H3Co(CN)6.

We investigate the structures of H(3)Co(CN)(6) and D(3)Co(CN)(6) and their temperature dependence using a combination of advanced neutron and x-ray diffraction techniques. Lattice parameter refinements show marked temperature- and isotope-dependent effects in the thermal expansion behaviour. Reverse Monte Carlo modelling of neutron total scattering data characterizes the distribution of D atom positions in the deuterated compound; analysis of these distributions reveals that the two N-H/D bonds become increasingly different with increasing temperature, suggesting a mechanism for the observed thermal expansion anomalies. We discuss these results in the context of the previous crystallographic, spectroscopic and theoretical results for H(3)Co(CN)(6) and 'super-short' N···H···N systems in general.

Use of expert judgment in exposure assessment: part 2. Calibration of expert judgments about personal exposures to benzene.

The recent movement of regulatory agencies toward probabilistic analyses of human health and environmental risks has focused greater attention on the quality of the estimates of variability and uncertainty that underlie them. Of particular concern is how uncertainty--a measure of what is not known--is characterized, as uncertainty can play an influential role in analyses of the need for regulatory controls or in estimates of the economic value of additional research. This paper reports the second phase of a study, conducted as an element of the National Human Exposure Assessment Survey (NHEXAS), to obtain and calibrate exposure assessment experts judgments about uncertainty in residential ambient, residential indoor, and personal air benzene concentrations experienced by the nonsmoking, nonoccupationally exposed population in U.S. EPA's Region V. Subjective judgments (i.e., the median, interquartile range, and 90% confidence interval) about the means and 90th percentiles of each of the benzene distributions were elicited from the seven experts participating in the study. The calibration or quality of the experts' judgments was assessed by comparing them to the actual measurements from the NHEXAS Region V study using graphical techniques, a quadratic scoring rule, and surprise and interquartile indices. The results from both quantitative scoring methods suggested that, considered collectively, the experts' judgments were relatively well calibrated although on balance, underconfident. The calibration of individual expert judgments appeared variable, highlighting potential pitfalls in reliance on individual experts. In a surprising finding, the experts' judgments about the 90th percentiles of the benzene distributions were better calibrated than their predictions about the means; the experts tended to be overconfident in their ability to predict the means. This paper is also one of the first calibration studies to demonstrate the importance of taking into account intraexpert correlation on the statistical significance of the findings. When the judgments were assumed to be independent, analysis of the surprise and interquartile indices found evidence of poor calibration (P<0.05). However, when the intraexpert correlation in the study was taken into account, these findings were no longer statistically significant. The analysis further found that the experts' judgments scored better than estimates of Region V benzene concentrations simply drawn from earlier studies of ambient, indoor and personal benzene levels in other U.S. cities. These results suggest the value of careful elicitation of expert judgments in characterizing exposures in probabilistic form. Additional calibration studies need to be undertaken to corroborate and extend these findings.

Policy uses of particulate exposure estimates.

This paper reviews the uses of particulate exposure estimates for science and policy. We propose a set of normative factors to guide the selection and application of various approaches for exposure assessment. For exposure estimates intended for use in support of the development or air pollution regulations or selection of control strategies, the proposed criteria include--compatibility with policy scope and scale, cost-effectiveness, characterization of uncertainty, political and institutional feasibility, and sensitivity to framing. For exposure assessment in support of epidemiological research, key criteria are-compatability with the specific hypothesis being tested, and compatibility with the temporal and spatial scale of analysis. The various major approaches for estimation of exposure--direct and indirect measurement, empirical and physical modeling--are catalogued and compared with these proposed criteria.