Detecting Hypoglycemia Incidents Reported in Patients' Secure Messages: Using Cost-Sensitive Learning and Oversampling to Reduce Data Imbalance.

BACKGROUND: Improper dosing of medications such as insulin can cause hypoglycemic episodes, which may lead to severe morbidity or even death. Although secure messaging was designed for exchanging nonurgent messages, patients sometimes report hypoglycemia events through secure messaging. Detecting these patient-reported adverse events may help alert clinical teams and enable early corrective actions to improve patient safety. OBJECTIVE: We aimed to develop a natural language processing system, called HypoDetect (Hypoglycemia Detector), to automatically identify hypoglycemia incidents reported in patients' secure messages. METHODS: An expert in public health annotated 3000 secure message threads between patients with diabetes and US Department of Veterans Affairs clinical teams as containing patient-reported hypoglycemia incidents or not. A physician independently annotated 100 threads randomly selected from this dataset to determine interannotator agreement. We used this dataset to develop and evaluate HypoDetect. HypoDetect incorporates 3 machine learning algorithms widely used for text classification: linear support vector machines, random forest, and logistic regression. We explored different learning features, including new knowledge-driven features. Because only 114 (3.80%) messages were annotated as positive, we investigated cost-sensitive learning and oversampling methods to mitigate the challenge of imbalanced data. RESULTS: The interannotator agreement was Cohen kappa=.976. Using cross-validation, logistic regression with cost-sensitive learning achieved the best performance (area under the receiver operating characteristic curve=0.954, sensitivity=0.693, specificity 0.974, F1 score=0.590). Cost-sensitive learning and the ensembled synthetic minority oversampling technique improved the sensitivity of the baseline systems substantially (by 0.123 to 0.728 absolute gains). Our results show that a variety of features contributed to the best performance of HypoDetect. CONCLUSIONS: Despite the challenge of data imbalance, HypoDetect achieved promising results for the task of detecting hypoglycemia incidents from secure messages. The system has a great potential to facilitate early detection and treatment of hypoglycemia.

Development and Validation of an Electronic Adverse Event Model for Patient Safety Surveillance in Interventional Radiology.

BACKGROUND: Comprehensive adverse event (AE) surveillance programs in interventional radiology (IR) are rare. Our aim was to develop and validate a retrospective electronic surveillance model to identify outpatient IR procedures that are likely to have an AE, to support patient safety and quality improvement. METHODS: We identified outpatient IR procedures performed in the period from October 2017 to September 2019 from the Veterans Health Administration (n = 135,283) and applied electronic triggers based on posyprocedure care to flag cases with a potential AE. From the trigger-flagged cases, we randomly sampled n = 1,500 for chart review to identify AEs. We also randomly sampled n = 600 from the unflagged cases. Chart-reviewed cases were merged with patient, procedure, and facility factors to estimate a mixed-effects logistic regression model designed to predict whether an AE occurred. Using model fit and criterion validity, we determined the best predicted probability threshold to identify cases with a likely AE. We reviewed a random sample of 200 cases above the threshold and 100 cases from below the threshold from October 2019 to March 2020 (n = 20,849) for model validation. RESULTS: In our development sample of mostly trigger-flagged cases, 444 of 2,096 cases (21.8%) had an AE. The optimal predicted probability threshold for a likely AE from our surveillance model was >50%, with positive predictive value of 68.9%, sensitivity of 38.3%, and specificity of 95.3%. In validation, chart-reviewed cases with AE probability >50% had a positive predictive value of 63% (n = 203). For the period from October 2017 to March 2020, the model identified approximately 70 IR cases per month that were likely to have an AE. CONCLUSIONS: This electronic trigger-based approach to AE surveillance could be used for patient-safety reporting and quality review.

Identifying Herbal Adverse Events From Spontaneous Reporting Systems Using Taxonomic Name Resolution Approach.

The efficacy and safety of herbal supplements suffer from challenges due to non-uniform representation of ingredient terms within biomedical and observational health data sources. The nature of how supplement data are reported within Spontaneous Reporting Systems (SRS) can limit analyses of supplement-associated adverse events due to the use of incorrect nomenclature or failing to identify herbs. This study aimed to extract, standardize, and summarize supplement-relevant reports from two SRSs: (1) Food and Drug Administration Adverse Event Reporting System (FAERS) and (2) Canada Vigilance Adverse Reaction (CVAR) database. A thesaurus of plant names was developed and integrated with a mapping and normalization approach that accommodated misspellings and variants. The reports gathered from FAERS between the years 2004 and 2016 show 185,915 herbal and 7,235,330 non-herbal accounting for 2.51%. The data from CVAR found 36,940 reports of herbal and 503,580 non-herbal reports between the years 1965 and 2017 for a total of 6.83%. Although not all cases were actual adverse events due to numerous variables and incomplete reporting, it is interesting to note that the herbs most frequently reported and significantly associated with adverse events were as follows: Avena sativa (Oats), Cannabis sativa (marijuana), Digitalis purpurea (foxglove), Humulus lupulus (hops), Hypericum perforatum (St John's Wort), Paullinia cupana (guarana), Phleum pretense (timothy-grass), Silybum marianum (milk thistle), Taraxacum officinale (Dandelion), and Valeriana officinalis (valerian). Using a scalable approach for mapping and resolution of herb names allowed data-driven exploration of potential adverse events from sources that have remained isolated in this specific area of research. The results from this study highlight several herb-associated safety issues providing motivation for subsequent in-depth analyses, including those that focus on the scope and severity of potential safety issues with supplement use.

Adverse event detection by integrating twitter data and VAERS.

BACKGROUND: Vaccine has been one of the most successful public health interventions to date. However, vaccines are pharmaceutical products that carry risks so that many adverse events (AEs) are reported after receiving vaccines. Traditional adverse event reporting systems suffer from several crucial challenges including poor timeliness. This motivates increasing social media-based detection systems, which demonstrate successful capability to capture timely and prevalent disease information. Despite these advantages, social media-based AE detection suffers from serious challenges such as labor-intensive labeling and class imbalance of the training data. RESULTS: To tackle both challenges from traditional reporting systems and social media, we exploit their complementary strength and develop a combinatorial classification approach by integrating Twitter data and the Vaccine Adverse Event Reporting System (VAERS) information aiming to identify potential AEs after influenza vaccine. Specifically, we combine formal reports which have accurately predefined labels with social media data to reduce the cost of manual labeling; in order to combat the class imbalance problem, a max-rule based multi-instance learning method is proposed to bias positive users. Various experiments were conducted to validate our model compared with other baselines. We observed that (1) multi-instance learning methods outperformed baselines when only Twitter data were used; (2) formal reports helped improve the performance metrics of our multi-instance learning methods consistently while affecting the performance of other baselines negatively; (3) the effect of formal reports was more obvious when the training size was smaller. Case studies show that our model labeled users and tweets accurately. CONCLUSIONS: We have developed a framework to detect vaccine AEs by combining formal reports with social media data. We demonstrate the power of formal reports on the performance improvement of AE detection when the amount of social media data was small. Various experiments and case studies show the effectiveness of our model.

Factors Associated with Hospital Admission after Outpatient Surgery in the Veterans Health Administration.

OBJECTIVE: To examine factors associated with 0- to 7-day admission after outpatient surgery in high-volume specialties: general surgery, orthopedics, urology, ear/nose/throat, and podiatry. STUDY DESIGN: We calculated rates and assessed diagnosis codes for 0- to 7-day admission after outpatient surgery for Centers for Medicare and Medicaid Services (CMS) and Veterans Health Administration (VA) dually enrolled patients age 65 and older. We also estimated separate multilevel logistic regression models to compare patient, procedure, and facility characteristics associated with postoperative admission. DATA COLLECTION: 2011-2013 surgical encounter data from the VA Corporate Data Warehouse; geographic data from the Area Health Resources File; CMS enrollment and hospital admission data. PRINCIPAL FINDINGS: Among 63,585 outpatient surgeries in 124 facilities, 0- to 7-day admission rates ranged from 5 percent (podiatry) to 28 percent (urology); nearly 66 percent of the admissions occurred on the day of surgery. Only 97 admissions were detected in the CMS data (1 percent). Surgical complications were diagnosed in 4 percent of admissions. Procedure complexity, measured by relative value units or anesthesia risk score, was associated with admission across all specialties. CONCLUSION: As many as 20 percent of VA outpatient surgeries result in an admission. Complex procedures are more likely to be followed by admission, but more evidence is required to determine how many of these reflect potential safety or quality problems.

Clinical criteria to screen for inpatient diagnostic errors: a scoping review.

Diagnostic errors are common and costly, but difficult to detect. "Trigger" tools have promise to facilitate detection, but have not been applied specifically for inpatient diagnostic error. We performed a scoping review to collate all individual "trigger" criteria that have been developed or validated that may indicate that an inpatient diagnostic error has occurred. We searched three databases and screened 8568 titles and abstracts to ultimately include 33 articles. We also developed a conceptual framework of diagnostic error outcomes using real clinical scenarios, and used it to categorize the extracted criteria. Of the multiple criteria we found related to inpatient diagnostic error and amenable to automated detection, the most common were death, transfer to a higher level of care, arrest or "code", and prolonged length of hospital stay. Several others, such as abrupt stoppage of multiple medications or change in procedure, may also be useful. Validation for general adverse event detection was done in 15 studies, but only one performed validation for diagnostic error specifically. Automated detection was used in only two studies. These criteria may be useful for developing diagnostic error detection tools.