Prospective validation of a machine learning model that uses provider notes to identify candidates for resective epilepsy surgery.

OBJECTIVE: Delay to resective epilepsy surgery results in avoidable disease burden and increased risk of mortality. The objective was to prospectively validate a natural language processing (NLP) application that uses provider notes to assign epilepsy surgery candidacy scores. METHODS: The application was trained on notes from (1) patients with a diagnosis of epilepsy and a history of resective epilepsy surgery and (2) patients who were seizure-free without surgery. The testing set included all patients with unknown surgical candidacy status and an upcoming neurology visit. Training and testing sets were updated weekly for 1 year. One- to three-word phrases contained in patients' notes were used as features. Patients prospectively identified by the application as candidates for surgery were manually reviewed by two epileptologists. Performance metrics were defined by comparing NLP-derived surgical candidacy scores with surgical candidacy status from expert chart review. RESULTS: The training set was updated weekly and included notes from a mean of 519 ± 67 patients. The area under the receiver operating characteristic curve (AUC) from 10-fold cross-validation was 0.90 ± 0.04 (range = 0.83-0.96) and improved by 0.002 per week (P < .001) as new patients were added to the training set. Of the 6395 patients who visited the neurology clinic, 4211 (67%) were evaluated by the model. The prospective AUC on this test set was 0.79 (95% confidence interval [CI] = 0.62-0.96). Using the optimal surgical candidacy score threshold, sensitivity was 0.80 (95% CI = 0.29-0.99), specificity was 0.77 (95% CI = 0.64-0.88), positive predictive value was 0.25 (95% CI = 0.07-0.52), and negative predictive value was 0.98 (95% CI = 0.87-1.00). The number needed to screen was 5.6. SIGNIFICANCE: An electronic health record-integrated NLP application can accurately assign surgical candidacy scores to patients in a clinical setting.

Identifying epilepsy psychiatric comorbidities with machine learning.

OBJECTIVE: People with epilepsy are at increased risk for mental health comorbidities. Machine-learning methods based on spoken language can detect suicidality in adults. This study's purpose was to use spoken words to create machine-learning classifiers that identify current or lifetime history of comorbid psychiatric conditions in teenagers and young adults with epilepsy. MATERIALS AND METHODS: Eligible participants were >12 years old with epilepsy. All participants were interviewed using the Mini International Neuropsychiatric Interview (MINI) or the MINI Kid Tracking and asked five open-ended conversational questions. N-grams and Linguistic Inquiry and Word Count (LIWC) word categories were used to construct machine learning classification models from language harvested from interviews. Data were analyzed for four individual MINI identified disorders and for three mutually exclusive groups: participants with no psychiatric disorders, participants with non-suicidal psychiatric disorders, and participants with any degree of suicidality. Performance was measured using areas under the receiver operating characteristic curve (AROCs). RESULTS: Classifiers were constructed from 227 interviews with 122 participants (7.5 ± 3.1 minutes and 454 ± 299 words). AROCs for models differentiating the non-overlapping groups and individual disorders ranged 57%-78% (many with P < .02). DISCUSSION AND CONCLUSION: Machine-learning classifiers of spoken language can reliably identify current or lifetime history of suicidality and depression in people with epilepsy. Data suggest identification of anxiety and bipolar disorders may be achieved with larger data sets. Machine-learning analysis of spoken language can be promising as a useful screening alternative when traditional approaches are unwieldy (eg, telephone calls, primary care offices, school health clinics).

The reliability of an epilepsy treatment clinical decision support system.

We developed a content validated computerized epilepsy treatment clinical decision support system to assist clinicians with selecting the best antiepilepsy treatments. Before disseminating our computerized epilepsy treatment clinical decision support system, further rigorous validation testing was necessary. As reliability is a precondition of validity, we verified proof of reliability first. We evaluated the consistency of the epilepsy treatment clinical decision support system in three areas including the preferred antiepilepsy drug choice, the top three recommended choices, and the rank order of the three choices. We demonstrated 100% reliability on 15,000 executions involving a three-step process on five different common pediatric epilepsy syndromes. Evidence for the reliability of the epilepsy treatment clinical decision support system was essential for the long-term viability of the system, and served as a crucial component for the next phase of system validation.

Epilepsy prevalence and severity predictors in MRI-identified focal cortical dysplasia.

OBJECTIVES: To determine the prevalence of epilepsy and drug-resistant epilepsy in pediatric patients with focal cortical dysplasia (FCD) identified by magnetic resonance imaging (MRI). To determine clinical and imaging differences between those with drug-resistant epilepsy, drug-responsive epilepsy, and no epilepsy among children with MRI-identified FCD. METHODS: A keyword search of a hospital radiology database identified 97 study participants for inclusion in this retrospective study. Participants were included if they were under 18 years of age at time of database query and had an MRI between 2004 and 2013 showing FCD. Exclusion was based on imaging and clinical characteristics. Data was gathered using a chart review and supplemental questionnaire. RESULTS: In this cohort of patients with imaging findings compatible with FCD, 29% had not developed epilepsy. The prevalence of epilepsy and drug-resistant epilepsy was 71.13% (95% C.I.=61.05-79.89%) and 32.99% (95% C.I.=23.78-43.27%), respectively. Patients with epilepsy were more likely to have temporal (p=0.029) or frontal (p=0.044) lobe lesions and a family history of seizures (p=0.003) than those without epilepsy. Age of seizure onset was later in those with drug-responsive epilepsy than those with drug-resistant epilepsy (p=0.0002). A later age of seizure onset (OR=1.22, p=0.0441, 95% C.I.=1.00-1.486) and absence of developmental delay (OR=3.624, p=0.0497, 95% C.I.=1.002-13.110) predicted a less severe epilepsy phenotype. CONCLUSIONS: Previous studies have only assessed patient cohorts with FCD and epilepsy, limiting the data on "asymptomatic" or "atypically presenting" FCD. Identifying a surprisingly large, novel cohort of children with FCD that had not developed epilepsy helps define prognosis and inform clinical management of children with FCD on imaging.

Methodological Issues in Predicting Pediatric Epilepsy Surgery Candidates Through Natural Language Processing and Machine Learning.

OBJECTIVE: We describe the development and evaluation of a system that uses machine learning and natural language processing techniques to identify potential candidates for surgical intervention for drug-resistant pediatric epilepsy. The data are comprised of free-text clinical notes extracted from the electronic health record (EHR). Both known clinical outcomes from the EHR and manual chart annotations provide gold standards for the patient's status. The following hypotheses are then tested: 1) machine learning methods can identify epilepsy surgery candidates as well as physicians do and 2) machine learning methods can identify candidates earlier than physicians do. These hypotheses are tested by systematically evaluating the effects of the data source, amount of training data, class balance, classification algorithm, and feature set on classifier performance. The results support both hypotheses, with F-measures ranging from 0.71 to 0.82. The feature set, classification algorithm, amount of training data, class balance, and gold standard all significantly affected classification performance. It was further observed that classification performance was better than the highest agreement between two annotators, even at one year before documented surgery referral. The results demonstrate that such machine learning methods can contribute to predicting pediatric epilepsy surgery candidates and reducing lag time to surgery referral.