Can Natural Language Processing Improve the Efficiency of Vaccine Adverse Event Report Review?

BACKGROUND: Individual case review of spontaneous adverse event (AE) reports remains a cornerstone of medical product safety surveillance for industry and regulators. Previously we developed the Vaccine Adverse Event Text Miner (VaeTM) to offer automated information extraction and potentially accelerate the evaluation of large volumes of unstructured data and facilitate signal detection. OBJECTIVE: To assess how the information extraction performed by VaeTM impacts the accuracy of a medical expert's review of the vaccine adverse event report. METHODS: The "outcome of interest" (diagnosis, cause of death, second level diagnosis), "onset time," and "alternative explanations" (drug, medical and family history) for the adverse event were extracted from 1000 reports from the Vaccine Adverse Event Reporting System (VAERS) using the VaeTM system. We compared the human interpretation, by medical experts, of the VaeTM extracted data with their interpretation of the traditional full text reports for these three variables. Two experienced clinicians alternately reviewed text miner output and full text. A third clinician scored the match rate using a predefined algorithm; the proportion of matches and 95% confidence intervals (CI) were calculated. Review time per report was analyzed. RESULTS: Proportion of matches between the interpretation of the VaeTM extracted data, compared to the interpretation of the full text: 93% for outcome of interest (95% CI: 91-94%) and 78% for alternative explanation (95% CI: 75-81%). Extracted data on the time to onset was used in 14% of cases and was a match in 54% (95% CI: 46-63%) of those cases. When supported by structured time data from reports, the match for time to onset was 79% (95% CI: 76-81%). The extracted text averaged 136 (74%) fewer words, resulting in a mean reduction in review time of 50 (58%) seconds per report. CONCLUSION: Despite a 74% reduction in words, the clinical conclusion from VaeTM extracted data agreed with the full text in 93% and 78% of reports for the outcome of interest and alternative explanation, respectively. The limited amount of extracted time interval data indicates the need for further development of this feature. VaeTM may improve review efficiency, but further study is needed to determine if this level of agreement is sufficient for routine use.

Simulating adverse event spontaneous reporting systems as preferential attachment networks: application to the Vaccine Adverse Event Reporting System.

BACKGROUND: Spontaneous Reporting Systems [SRS] are critical tools in the post-licensure evaluation of medical product safety. Regulatory authorities use a variety of data mining techniques to detect potential safety signals in SRS databases. Assessing the performance of such signal detection procedures requires simulated SRS databases, but simulation strategies proposed to date each have limitations. OBJECTIVE: We sought to develop a novel SRS simulation strategy based on plausible mechanisms for the growth of databases over time. METHODS: We developed a simulation strategy based on the network principle of preferential attachment. We demonstrated how this strategy can be used to create simulations based on specific databases of interest, and provided an example of using such simulations to compare signal detection thresholds for a popular data mining algorithm. RESULTS: The preferential attachment simulations were generally structurally similar to our targeted SRS database, although they had fewer nodes of very high degree. The approach was able to generate signal-free SRS simulations, as well as mimicking specific known true signals. Explorations of different reporting thresholds for the FDA Vaccine Adverse Event Reporting System suggested that using proportional reporting ratio [PRR] > 3.0 may yield better signal detection operating characteristics than the more commonly used PRR > 2.0 threshold. DISCUSSION: The network analytic approach to SRS simulation based on the principle of preferential attachment provides an attractive framework for exploring the performance of safety signal detection algorithms. This approach is potentially more principled and versatile than existing simulation approaches. CONCLUSION: The utility of network-based SRS simulations needs to be further explored by evaluating other types of simulated signals with a broader range of data mining approaches, and comparing network-based simulations with other simulation strategies where applicable.

The contribution of the vaccine adverse event text mining system to the classification of possible Guillain-Barré syndrome reports.

BACKGROUND: We previously demonstrated that a general purpose text mining system, the Vaccine adverse event Text Mining (VaeTM) system, could be used to automatically classify reports of an-aphylaxis for post-marketing safety surveillance of vaccines. OBJECTIVE: To evaluate the ability of VaeTM to classify reports to the Vaccine Adverse Event Reporting System (VAERS) of possible Guillain-Barré Syndrome (GBS). METHODS: We used VaeTM to extract the key diagnostic features from the text of reports in VAERS. Then, we applied the Brighton Collaboration (BC) case definition for GBS, and an information retrieval strategy (i.e. the vector space model) to quantify the specific information that is included in the key features extracted by VaeTM and compared it with the encoded information that is already stored in VAERS as Medical Dictionary for Regulatory Activities (MedDRA) Preferred Terms (PTs). We also evaluated the contribution of the primary (diagnosis and cause of death) and secondary (second level diagnosis and symptoms) diagnostic VaeTM-based features to the total VaeTM-based information. RESULTS: MedDRA captured more information and better supported the classification of reports for GBS than VaeTM (AUC: 0.904 vs. 0.777); the lower performance of VaeTM is likely due to the lack of extraction by VaeTM of specific laboratory results that are included in the BC criteria for GBS. On the other hand, the VaeTM-based classification exhibited greater specificity than the MedDRA-based approach (94.96% vs. 87.65%). Most of the VaeTM-based information was contained in the secondary diagnostic features. CONCLUSION: For GBS, clinical signs and symptoms alone are not sufficient to match MedDRA coding for purposes of case classification, but are preferred if specificity is the priority.

Automating case definitions using literature-based reasoning.

BACKGROUND: Establishing a Case Definition (CDef) is a first step in many epidemiological, clinical, surveillance, and research activities. The application of CDefs still relies on manual steps and this is a major source of inefficiency in surveillance and research. OBJECTIVE: Describe the need and propose an approach for automating the useful representation of CDefs for medical conditions. METHODS: We translated the existing Brighton Collaboration CDef for anaphylaxis by mostly relying on the identification of synonyms for the criteria of the CDef using the NLM MetaMap tool. We also generated a CDef for the same condition using all the related PubMed abstracts, processing them with a text mining tool, and further treating the synonyms with the above strategy. The co-occurrence of the anaphylaxis and any other medical term within the same sentence of the abstracts supported the construction of a large semantic network. The 'islands' algorithm reduced the network and revealed its densest region including the nodes that were used to represent the key criteria of the CDef. We evaluated the ability of the "translated" and the "generated" CDef to classify a set of 6034 H1N1 reports for anaphylaxis using two similarity approaches and comparing them with our previous semi-automated classification approach. RESULTS: Overall classification performance across approaches to producing CDefs was similar, with the generated CDef and vector space model with cosine similarity having the highest accuracy (0.825 ± 0.003) and the semi-automated approach and vector space model with cosine similarity having the highest recall (0.809 ± 0.042). Precision was low for all approaches. CONCLUSION: The useful representation of CDefs is a complicated task but potentially offers substantial gains in efficiency to support safety and clinical surveillance.

Can network analysis improve pattern recognition among adverse events following immunization reported to VAERS?

Current methods of statistical data mining are limited in their ability to facilitate the identification of patterns of potential clinical interest from spontaneous reporting systems of medical product adverse events (AEs). Network analysis (NA) allows for simultaneous representation of complex connections among the key elements of such a system. The Vaccine Adverse Event Reporting System (VAERS) can be represented as a network of 6,428 nodes (74 vaccines and 6,354 AEs) with more than 1.4 million interlinkages. VAERS has the characteristics of a "scale-free" network, with certain vaccines and AEs acting as "hubs" in the network. Known safety signals were visualized using NA methods, including hub identification. NA offers a complementary approach to current statistical data-mining techniques for visualizing multidimensional patterns, providing a structural framework for evaluating AE data.

Neural networks for the prediction of spirometric reference values.

Normal lung function values are conventionally calculated according to prediction equations. The primary objective of this study is the development of a different method for the prediction of FVC and FEV1 parameters, in order to achieve better correlation of the predicted values to the real ones. Using a sample from the Greek elderly population that was separated into two groups (a training and a testing one), a number of artificial neural networks were trained. Considering that men and women were studied separately and that two parameters (FVC, FEV1) were the target of the study, four cases came up. In each case two neural networks were trained using different transfer functions, number of neurons and number of layers. When passing the inputs of the testing data set to the trained networks it was found that the outputs were well correlated with the corresponding measures of the sample. Furthermore, the match with the sample, for a number of neural networks developed, was better compared to the matches of Baltopoulos et al. study that used the same sample for developing prediction equations. This high match allows the potential use of neural networks for predicting not only FVC and FEV1 but also other spirometric parameters.