Prospective validation of a seizure diary forecasting falls short.

ABSTRACT

OBJECTIVE: Recently, a deep learning artificial intelligence (AI) model forecasted seizure risk using retrospective seizure diaries with higher accuracy than random forecasts. The present study sought to prospectively evaluate the same algorithm. METHODS: We recruited a prospective cohort of 46 people with epilepsy; 25 completed sufficient data entry for analysis (median = 5 months). We used the same AI method as in our prior study. Group-level and individual-level Brier Skill Scores (BSSs) compared random forecasts and simple moving average forecasts to the AI. RESULTS: The AI had an area under the receiver operating characteristic curve of .82. At the group level, the AI outperformed random forecasting (BSS = .53). At the individual level, AI outperformed random in 28% of cases. At the group and individual level, the moving average outperformed the AI. If pre-enrollment (nonverified) diaries (with presumed underreporting) were included, the AI significantly outperformed both comparators. Surveys showed most did not mind poor-quality LOW-RISK or HIGH-RISK forecasts, yet 91% wanted access to these forecasts. SIGNIFICANCE: The previously developed AI forecasting tool did not outperform a very simple moving average forecasting in this prospective cohort, suggesting that the AI model should be replaced.