Evaluation of an open forecasting challenge to assess skill of West Nile virus neuroinvasive disease prediction.

BACKGROUND: West Nile virus (WNV) is the leading cause of mosquito-borne illness in the continental USA. WNV occurrence has high spatiotemporal variation, and current approaches to targeted control of the virus are limited, making forecasting a public health priority. However, little research has been done to compare strengths and weaknesses of WNV disease forecasting approaches on the national scale. We used forecasts submitted to the 2020 WNV Forecasting Challenge, an open challenge organized by the Centers for Disease Control and Prevention, to assess the status of WNV neuroinvasive disease (WNND) prediction and identify avenues for improvement. METHODS: We performed a multi-model comparative assessment of probabilistic forecasts submitted by 15 teams for annual WNND cases in US counties for 2020 and assessed forecast accuracy, calibration, and discriminatory power. In the evaluation, we included forecasts produced by comparison models of varying complexity as benchmarks of forecast performance. We also used regression analysis to identify modeling approaches and contextual factors that were associated with forecast skill. RESULTS: Simple models based on historical WNND cases generally scored better than more complex models and combined higher discriminatory power with better calibration of uncertainty. Forecast skill improved across updated forecast submissions submitted during the 2020 season. Among models using additional data, inclusion of climate or human demographic data was associated with higher skill, while inclusion of mosquito or land use data was associated with lower skill. We also identified population size, extreme minimum winter temperature, and interannual variation in WNND cases as county-level characteristics associated with variation in forecast skill. CONCLUSIONS: Historical WNND cases were strong predictors of future cases with minimal increase in skill achieved by models that included other factors. Although opportunities might exist to specifically improve predictions for areas with large populations and low or high winter temperatures, areas with high case-count variability are intrinsically more difficult to predict. Also, the prediction of outbreaks, which are outliers relative to typical case numbers, remains difficult. Further improvements to prediction could be obtained with improved calibration of forecast uncertainty and access to real-time data streams (e.g. current weather and preliminary human cases).

Evaluation of FluSight influenza forecasting in the 2021-22 and 2022-23 seasons with a new target laboratory-confirmed influenza hospitalizations.

Accurate forecasts can enable more effective public health responses during seasonal influenza epidemics. Forecasting teams were asked to provide national and jurisdiction-specific probabilistic predictions of weekly confirmed influenza hospital admissions for one through four weeks ahead for the 2021-22 and 2022-23 influenza seasons. Across both seasons, 26 teams submitted forecasts, with the submitting teams varying between seasons. Forecast skill was evaluated using the Weighted Interval Score (WIS), relative WIS, and coverage. Six out of 23 models outperformed the baseline model across forecast weeks and locations in 2021-22 and 12 out of 18 models in 2022-23. Averaging across all forecast targets, the FluSight ensemble was the 2nd most accurate model measured by WIS in 2021-22 and the 5th most accurate in the 2022-23 season. Forecast skill and 95% coverage for the FluSight ensemble and most component models degraded over longer forecast horizons and during periods of rapid change. Current influenza forecasting efforts help inform situational awareness, but research is needed to address limitations, including decreased performance during periods of changing epidemic dynamics.

West Nile Virus and Other Domestic Nationally Notifiable Arboviral Diseases - United States, 2019.

Arthropod-borne viruses (arboviruses) are transmitted to humans primarily through the bites of infected mosquitoes and ticks. West Nile virus (WNV) is the leading cause of domestically acquired arboviral disease in the United States (1). Other arboviruses, including La Crosse, Jamestown Canyon, Powassan, eastern equine encephalitis, and St. Louis encephalitis viruses, cause sporadic disease and occasional outbreaks. This report summarizes surveillance data for nationally notifiable domestic arboviruses reported to CDC for 2019. For 2019, 47 states and the District of Columbia (DC) reported 1,173 cases of domestic arboviral disease, including 971 (83%) WNV disease cases. Among the WNV disease cases, 633 (65%) were classified as neuroinvasive disease, for a national incidence of 0.19 cases per 100,000 population, 53% lower than the median annual incidence during 2009-2018. More Powassan and eastern equine encephalitis virus disease cases were reported in 2019 than in any previous year. Health care providers should consider arboviral infections in patients with aseptic meningitis or encephalitis, perform recommended diagnostic testing, and promptly report cases to public health authorities. Because arboviral diseases continue to cause serious illness, and annual incidence of individual viruses continues to vary with sporadic outbreaks, maintaining surveillance is important in directing prevention activities. Prevention depends on community and household efforts to reduce vector populations and personal protective measures to prevent mosquito and tick bites such as use of Environmental Protection Agency-registered insect repellent and wearing protective clothing.*,†.

Surveillance for West Nile virus disease - United States, 2009-2018.

PROBLEM/CONDITION: West Nile virus (WNV) is an arthropod-borne virus (arbovirus) in the family Flaviviridae and is the leading cause of domestically acquired arboviral disease in the contiguous United States. An estimated 70%-80% of WNV infections are asymptomatic. Symptomatic persons usually develop an acute systemic febrile illness. Less than 1% of infected persons develop neuroinvasive disease, which typically presents as encephalitis, meningitis, or acute flaccid paralysis. REPORTING PERIOD: 2009-2018. DESCRIPTION OF SYSTEM: WNV disease is a nationally notifiable condition with standard surveillance case definitions. State health departments report WNV cases to CDC through ArboNET, an electronic passive surveillance system. Variables collected include patient age, sex, race, ethnicity, county and state of residence, date of illness onset, clinical syndrome, hospitalization, and death. RESULTS: During 2009-2018, a total of 21 869 confirmed or probable cases of WNV disease, including 12 835 (59%) WNV neuroinvasive disease cases, were reported to CDC from all 50 states, the District of Columbia, and Puerto Rico. A total of 89% of all WNV patients had illness onset during July-September. Neuroinvasive disease incidence and case-fatalities increased with increasing age, with the highest incidence (1.22 cases per 100 000 population) occurring among persons aged ≥70 years. Among neuroinvasive cases, hospitalization rates were >85% in all age groups but were highest among patients aged ≥70 years (98%). The national incidence of WNV neuroinvasive disease peaked in 2012 (0.92 cases per 100 000 population). Although national incidence was relatively stable during 2013-2018 (average annual incidence: 0.44; range: 0.40-0.51), state level incidence varied from year to year. During 2009-2018, the highest average annual incidence of neuroinvasive disease occurred in North Dakota (3.16 cases per 100 000 population), South Dakota (3.06), Nebraska (1.95), and Mississippi (1.17), and the largest number of total cases occurred in California (2819), Texas (2043), Illinois (728), and Arizona (632). Six counties located within the four states with the highest case counts accounted for 23% of all neuroinvasive disease cases nationally. INTERPRETATION: Despite the recent stability in annual national incidence of neuroinvasive disease, peaks in activity were reported in different years for different regions of the country. Variations in vectors, avian amplifying hosts, human activity, and environmental factors make it difficult to predict future WNV disease incidence and outbreak locations. PUBLIC HEALTH ACTION: WNV disease surveillance is important for detecting and monitoring seasonal epidemics and for identifying persons at increased risk for severe disease. Surveillance data can be used to inform prevention and control activities. Health care providers should consider WNV infection in the differential diagnosis of aseptic meningitis and encephalitis, obtain appropriate specimens for testing, and promptly report cases to public health authorities. Public health education programs should focus prevention messaging on older persons because they are at increased risk for severe neurologic disease and death. In the absence of a human vaccine, WNV disease prevention depends on community-level mosquito control and household and personal protective measures. Understanding the geographic distribution of cases, particularly at the county level, appears to provide the best opportunity for directing finite resources toward effective prevention and control activities. Additional work to further develop and improve predictive models that can foreshadow areas most likely to be impacted in a given year by WNV outbreaks could allow for proactive targeting of interventions and ultimately lowering of WNV disease morbidity and mortality.

Surveillance for West Nile Virus Disease - United States, 2009-2018.

PROBLEM/CONDITION: West Nile virus (WNV) is an arthropodborne virus (arbovirus) in the family Flaviviridae and is the leading cause of domestically acquired arboviral disease in the contiguous United States. An estimated 70%-80% of WNV infections are asymptomatic. Symptomatic persons usually develop an acute systemic febrile illness. Less than 1% of infected persons develop neuroinvasive disease, which typically presents as encephalitis, meningitis, or acute flaccid paralysis. REPORTING PERIOD: 2009-2018. DESCRIPTION OF SYSTEM: WNV disease is a nationally notifiable condition with standard surveillance case definitions. State health departments report WNV cases to CDC through ArboNET, an electronic passive surveillance system. Variables collected include patient age, sex, race, ethnicity, county and state of residence, date of illness onset, clinical syndrome, hospitalization, and death. RESULTS: During 2009-2018, a total of 21,869 confirmed or probable cases of WNV disease, including 12,835 (59%) WNV neuroinvasive disease cases, were reported to CDC from all 50 states, the District of Columbia, and Puerto Rico. A total of 89% of all WNV patients had illness onset during July-September. Neuroinvasive disease incidence and case-fatalities increased with increasing age, with the highest incidence (1.22 cases per 100,000 population) occurring among persons aged ≥70 years. Among neuroinvasive cases, hospitalization rates were >85% in all age groups but were highest among patients aged ≥70 years (98%). The national incidence of WNV neuroinvasive disease peaked in 2012 (0.92 cases per 100,000 population). Although national incidence was relatively stable during 2013-2018 (average annual incidence: 0.44; range: 0.40-0.51), state level incidence varied from year to year. During 2009-2018, the highest average annual incidence of neuroinvasive disease occurred in North Dakota (3.16 cases per 100,000 population), South Dakota (3.06), Nebraska (1.95), and Mississippi (1.17), and the largest number of total cases occurred in California (2,819), Texas (2,043), Illinois (728), and Arizona (632). Six counties located within the four states with the highest case counts accounted for 23% of all neuroinvasive disease cases nationally. INTERPRETATION: Despite the recent stability in annual national incidence of neuroinvasive disease, peaks in activity were reported in different years for different regions of the country. Variations in vectors, avian amplifying hosts, human activity, and environmental factors make it difficult to predict future WNV disease incidence and outbreak locations. PUBLIC HEALTH ACTION: WNV disease surveillance is important for detecting and monitoring seasonal epidemics and for identifying persons at increased risk for severe disease. Surveillance data can be used to inform prevention and control activities. Health care providers should consider WNV infection in the differential diagnosis of aseptic meningitis and encephalitis, obtain appropriate specimens for testing, and promptly report cases to public health authorities. Public health education programs should focus prevention messaging on older persons, because they are at increased risk for severe neurologic disease and death. In the absence of a human vaccine, WNV disease prevention depends on community-level mosquito control and household and personal protective measures. Understanding the geographic distribution of cases, particularly at the county level, appears to provide the best opportunity for directing finite resources toward effective prevention and control activities. Additional work to further develop and improve predictive models that can foreshadow areas most likely to be impacted in a given year by WNV outbreaks could allow for proactive targeting of interventions and ultimately lowering of WNV disease morbidity and mortality.