Evaluation of Natural Language Processing (NLP) systems to annotate drug product labeling with MedDRA terminology.

INTRODUCTION: The FDA Adverse Event Reporting System (FAERS) is a primary data source for identifying unlabeled adverse events (AEs) in a drug or biologic drug product's postmarketing phase. Many AE reports must be reviewed by drug safety experts to identify unlabeled AEs, even if the reported AEs are previously identified, labeled AEs. Integrating the labeling status of drug product AEs into FAERS could increase report triage and review efficiency. Medical Dictionary for Regulatory Activities (MedDRA) is the standard for coding AE terms in FAERS cases. However, drug manufacturers are not required to use MedDRA to describe AEs in product labels. We hypothesized that natural language processing (NLP) tools could assist in automating the extraction and MedDRA mapping of AE terms in drug product labels. MATERIALS AND METHODS: We evaluated the performance of three NLP systems, (ETHER, I2E, MetaMap) for their ability to extract AE terms from drug labels and translate the terms to MedDRA Preferred Terms (PTs). Pharmacovigilance-based annotation guidelines for extracting AE terms from drug labels were developed for this study. We compared each system's output to MedDRA PT AE lists, manually mapped by FDA pharmacovigilance experts using the guidelines, for ten drug product labels known as the "gold standard AE list" (GSL) dataset. Strict time and configuration conditions were imposed in order to test each system's capabilities under conditions of no human intervention and minimal system configuration. Each NLP system's output was evaluated for precision, recall and F measure in comparison to the GSL. A qualitative error analysis (QEA) was conducted to categorize a random sample of each NLP system's false positive and false negative errors. RESULTS: A total of 417, 278, and 250 false positive errors occurred in the ETHER, I2E, and MetaMap outputs, respectively. A total of 100, 80, and 187 false negative errors occurred in ETHER, I2E, and MetaMap outputs, respectively. Precision ranged from 64% to 77%, recall from 64% to 83% and F measure from 67% to 79%. I2E had the highest precision (77%), recall (83%) and F measure (79%). ETHER had the lowest precision (64%). MetaMap had the lowest recall (64%). The QEA found that the most prevalent false positive errors were context errors such as "Context error/General term", "Context error/Instructions or monitoring parameters", "Context error/Medical history preexisting condition underlying condition risk factor or contraindication", and "Context error/AE manifestations or secondary complication". The most prevalent false negative errors were in the "Incomplete or missed extraction" error category. Missing AE terms were typically due to long terms, or terms containing non-contiguous words which do not correspond exactly to MedDRA synonyms. MedDRA mapping errors were a minority of errors for ETHER and I2E but were the most prevalent false positive errors for MetaMap. CONCLUSIONS: The results demonstrate that it may be feasible to use NLP tools to extract and map AE terms to MedDRA PTs. However, the NLP tools we tested would need to be modified or reconfigured to lower the error rates to support their use in a regulatory setting. Tools specific for extracting AE terms from drug labels and mapping the terms to MedDRA PTs may need to be developed to support pharmacovigilance. Conducting research using additional NLP systems on a larger, diverse GSL would also be informative.

Changes in US antidepressant and antipsychotic prescription patterns during a period of FDA actions.

PURPOSE: To determine if paroxetine versus non-paroxetine selective serotonin reuptake inhibitors (SSRIs) prescribing changed after the June 2003 FDA Paroxetine Public Health Advisory (PPHA) and if antidepressant and antipsychotic prescribing changed after the February 2004 FDA Advisory Committee Meeting (FDACM). METHODS: Ecologic analysis using estimates of patients dispensed antidepressants and antipsychotics, census data, and promotional spending data. Data sources were SDI: Vector One(R), US Census, and IMS Health(R). Measures were monthly use levels (number of patients dispensed antidepressants, antipsychotics, paroxetine, and non-paroxetine SSRIs prescriptions by age group per population count). Percent changes pre- to post-PPHA were used to assess changes in paroxetine versus non-paroxetine SSRIs prescribing. Interrupted time series (ITS) analysis was performed to examine use level changes post-FDACM by drug groups (all antidepressants and all antipsychotics). RESULTS: Post-PPHA mean paroxetine use levels decreased for all age groups (range: 5.5-34.1%). Mean non-paroxetine SSRIs use levels increased (range: 4.6-17.1%). Post-PPHA changes were greatest for 6-12 and 13-17 year olds. Decreased mean antidepressant drug use levels from pre- to post-FDACM were observed in all groups under 25 years old. A statistically significant decrease in the slopes from pre- to post-FDACM was observed for persons aged 13-17 and 18-24 years. The difference between the forecasted mean use level and the observed mean use level (in 12-month intervals) was statistically significant for all ages combined (-107.26; 95% CI: -166.32, -48.20) and 1-5 (-3.1; 95% CI: -4.62, -1.58), 6-12 (-36.02; 95% CI: -62.92, -9.12) and 25 years, and older groups (-83.17; 95% CI: -153.95, -12.39). For all age groups, decreases in the slopes of antipsychotic drugs use from pre- to post-FDACM were observed, although these slope changes were not statistically significant. The difference between the forecasted mean antipsychotic drugs use level and the observed mean use level (in 12-month intervals) was statistically significantly lower for all age groups. CONCLUSIONS: Antidepressant use changed post-PPHA and -FDACM, with a differential pattern by age. There was no evidence of increased antipsychotic use post-FDACM. Ecologic data cannot determine if changes were due to depression not treated with medications or the prescribing of fewer antidepressants for other conditions.

Validating a Framework for Coding Patient-Reported Health Information to the Medical Dictionary for Regulatory Activities Terminology: An Evaluative Study.

BACKGROUND: The availability of and interest in patient-generated health data (PGHD) have grown steadily. Patients describe medical experiences differently compared with how clinicians or researchers would describe their observations of those same experiences. Patients may find nonserious, known adverse drug events (ADEs) to be an ongoing concern, which impacts the tolerability and adherence. Clinicians must be vigilant for medically serious, potentially fatal ADEs. Having both perspectives provides patients and clinicians with a complete picture of what to expect from drug therapies. Multiple initiatives seek to incorporate patients' perspectives into drug development, including PGHD exploration for pharmacovigilance. The Food and Drug Administration (FDA) Adverse Event Reporting System contains case reports of postmarketing ADEs. To facilitate the analysis of these case reports, case details are coded using the Medical Dictionary for Regulatory Activities (MedDRA). PatientsLikeMe is a Web-based network where patients report, track, share, and discuss their health information. PatientsLikeMe captures PGHD through free-text and structured data fields. PatientsLikeMe structured data are coded to multiple medical terminologies, including MedDRA. The standardization of PatientsLikeMe PGHD enables electronic accessibility and enhances patient engagement. OBJECTIVE: The aim of this study is to retrospectively review PGHD for symptoms and ADEs entered by patients on PatientsLikeMe and coded by PatientsLikeMe to MedDRA terminology for concordance with regulatory-focused coding practices. METHODS: An FDA MedDRA coding expert retrospectively reviewed a data file containing verbatim patient-reported symptoms and ADEs and PatientsLikeMe-assigned MedDRA terms to determine the medical accuracy and appropriateness of the selected MedDRA terms, applying the International Council for Harmonisation MedDRA Term Selection: Points to Consider (MTS:PTC) guides. RESULTS: The FDA MedDRA coding expert reviewed 3234 PatientsLikeMe-assigned MedDRA codes and patient-reported verbatim text. The FDA and PatientsLikeMe were concordant at 97.09% (3140/3234) of the PatientsLikeMe-assigned MedDRA codes. The 2.91% (94/3234) discordant subset was analyzed to identify reasons for differences. Coding differences were attributed to several reasons but mostly driven by PatientsLikeMe's approach of assigning a more general MedDRA term to enable patient-to-patient engagement, while the FDA assigned a more specific medically relevant term. CONCLUSIONS: PatientsLikeMe MedDRA coding of PGHD was generally comparable to how the FDA would code similar data, applying the MTS:PTC principles. Discordant coding resulted from several reasons but mostly reflected a difference in purpose. The MTS:PTC coding principles aim to capture the most specific reported information about an ADE, whereas PatientsLikeMe may code patient-reported symptoms and ADEs to more general MedDRA terms to support patient engagement among a larger group of patients. This study demonstrates that most verbatim reports of symptoms and ADEs collected by a PGHD source, such as the PatientsLikeMe platform, could be reliably coded to MedDRA terminology by applying the MTS:PTC guide. Regarding all secondary use of novel data, understanding coding and standardization principles applied to these data types are important.

Development of an automated assessment tool for MedWatch reports in the FDA adverse event reporting system.

OBJECTIVE: As the US Food and Drug Administration (FDA) receives over a million adverse event reports associated with medication use every year, a system is needed to aid FDA safety evaluators in identifying reports most likely to demonstrate causal relationships to the suspect medications. We combined text mining with machine learning to construct and evaluate such a system to identify medication-related adverse event reports. METHODS: FDA safety evaluators assessed 326 reports for medication-related causality. We engineered features from these reports and constructed random forest, L1 regularized logistic regression, and support vector machine models. We evaluated model accuracy and further assessed utility by generating report rankings that represented a prioritized report review process. RESULTS: Our random forest model showed the best performance in report ranking and accuracy, with an area under the receiver operating characteristic curve of 0.66. The generated report ordering assigns reports with a higher probability of medication-related causality a higher rank and is significantly correlated to a perfect report ordering, with a Kendall's tau of 0.24 ( P = .002). CONCLUSION: Our models produced prioritized report orderings that enable FDA safety evaluators to focus on reports that are more likely to contain valuable medication-related adverse event information. Applying our models to all FDA adverse event reports has the potential to streamline the manual review process and greatly reduce reviewer workload.

Evaluation of Facebook and Twitter Monitoring to Detect Safety Signals for Medical Products: An Analysis of Recent FDA Safety Alerts.

INTRODUCTION: The rapid expansion of the Internet and computing power in recent years has opened up the possibility of using social media for pharmacovigilance. While this general concept has been proposed by many, central questions remain as to whether social media can provide earlier warnings for rare and serious events than traditional signal detection from spontaneous report data. OBJECTIVE: Our objective was to examine whether specific product-adverse event pairs were reported via social media before being reported to the US FDA Adverse Event Reporting System (FAERS). METHODS: A retrospective analysis of public Facebook and Twitter data was conducted for 10 recent FDA postmarketing safety signals at the drug-event pair level with six negative controls. Social media data corresponding to two years prior to signal detection of each product-event pair were compiled. Automated classifiers were used to identify each 'post with resemblance to an adverse event' (Proto-AE), among English language posts. A custom dictionary was used to translate Internet vernacular into Medical Dictionary for Regulatory Activities (MedDRA®) Preferred Terms. Drug safety physicians conducted a manual review to determine causality using World Health Organization-Uppsala Monitoring Centre (WHO-UMC) assessment criteria. Cases were also compared with those reported in FAERS. FINDINGS: A total of 935,246 posts were harvested from Facebook and Twitter, from March 2009 through October 2014. The automated classifier identified 98,252 Proto-AEs. Of these, 13 posts were selected for causality assessment of product-event pairs. Clinical assessment revealed that posts had sufficient information to warrant further investigation for two possible product-event associations: dronedarone-vasculitis and Banana Boat Sunscreen--skin burns. No product-event associations were found among the negative controls. In one of the positive cases, the first report occurred in social media prior to signal detection from FAERS, whereas the other case occurred first in FAERS. CONCLUSIONS: An efficient semi-automated approach to social media monitoring may provide earlier insights into certain adverse events. More work is needed to elaborate additional uses for social media data in pharmacovigilance and to determine how they can be applied by regulatory agencies.

Postmarketing surveillance of suicidal adverse events with pediatric use of antidepressants.

OBJECTIVE: The aim of this analysis was to delineate trends in spontaneous postmarketing reporting data with antidepressant drugs for adverse events involving suicidal behaviors in children and adolescents. METHODS: The U.S. Food and Drug Administration (FDA) Adverse Event Reporting System (AERS) was searched for postmarketing adverse event reports of suicidal thoughts and behaviors occurring in children and adolescents treated with 10 antidepressant drugs. The search covered the period from market launch of each drug through November 2003. RESULTS: A total of 524 reports were returned by the search. All drugs had reports, and most drugs demonstrated 15 or fewer reports annually, with the following two exceptions. We observed a peak of reporting for fluoxetine in the early 1990s, and another peak of reporting for paroxetine in recent years. Further investigation revealed that the peak in recent paroxetine reporting was accounted for by reports from consumers, whereas reporting by health professionals remained fairly constant. In contrast, the earlier peak in reports for fluoxetine was not accounted for by an increase in consumer reporting. CONCLUSIONS: Spontaneous reporting data for suicidal events in pediatric patients treated with antidepressant drugs appears to be highly variable and subject to various influences. The most appropriate method to assess an association of antidepressant drug treatment with suicidal behaviors is examination of systematically collected data with appropriate comparison groups, such as randomized, controlled trial data.

Developing and Initiating Validation of a Model Opioid Patient-Prescriber Agreement as a Tool for Patient-Centered Pain Treatment.

BACKGROUND: Opioid treatment agreements generally are used in pain treatment to delineate the terms and consequences of opioid use and abuse. METHODS: The US Food and Drug Administration (FDA) Safe Use Initiative convened a multi-disciplinary working group with outside experts to draft a patient-centered, model opioid treatment agreement named the Model Patient-Prescriber Agreement (model PPA). The model PPA was evaluated for readability and usability in two tests that sampled both healthcare professional and non-healthcare professional FDA employees. In a survey sent to FDA employees in the Center for Drug Evaluation and Research (CDER), 209 respondents assessed the quality of the content and the level of difficulty in reading and understanding the model PPA. Ten other FDA employees participated in usability testing to assess the effectiveness of the model PPA as an educational and decision-making tool. RESULTS: The majority of the 209 CDER employee survey respondents indicated the model PPA was neutral in tone (67.5%) and easy or somewhat easy to understand (90.4%). Usability study participants generally thought the model PPA would facilitate discussion between patient and prescriber and that the content was informative, thorough, and clear. CONCLUSIONS: These studies suggest that the working group was able to develop an opioid PPA that may be acceptable and usable among a diverse population of stakeholders. A follow-up pilot study using the model PPA in medical facilities in the USA with patients is underway and will facilitate this determination.

New antiarrhythmic agents for atrial fibrillation and atrial flutter: United States drug market response as an indicator of acceptance.

OBJECTIVE: To determine how well dofetilide and Betapace AF (sotalol, approved solely for atrial fibrillation and atrial flutter), with their detailed dosing and monitoring guidelines for safety, were accepted into clinical practice during the 2 calendar years after their introduction. METHODS AND RESULTS: We reviewed the number of new, refill, and total prescriptions of all antiarrhythmic agents in the United States from April 2000-December 2001 to assess use of dofetilide and Betapace AF in the drug market. Both were prescribed very infrequently throughout the study period. In addition, the infrequent reported use of these drugs for patients with atrial fibrillation and flutter indicated poor acceptance of these agents by prescribing physicians. We speculated that the restricted distribution and required educational program for dofetilide, as well as the availability of generic sotalol products, may have discouraged physicians from prescribing both dofetilide and Betapace AE CONCLUSION: A common goal for both the dofetilide risk-management program and the creation of a sotalol product indicated solely for atrial fibrillation and atrial flutter was to provide safer treatment for patients with these arrhythmias. Unfortunately, limited penetration of dofetilide and Betapace AF into the U.S. market suggests that drugs without a risk-management program or detailed dosing guidelines were more likely than dofetilide or Betapace AF to be selected for treatment of atrial fibrillation and atrial flutter.

Analysis of Maryland poisoning deaths using classification and regression tree (CART) analysis.

Our study is a cross-sectional analysis of Maryland poisoning deaths for years 2003 and 2004. We used Classification and Regression Tree (CART) methodology to classify undetermined intent Maryland poisoning deaths as either unintentional or suicidal poisonings. The predictive ability of the selected set of variables (i.e., poisoned in the home or workplace, location type, where poisoned, place of death, poison type, victim race and age, year of death) was extremely good. Of the 301 test cases, only eight were misclassified by the CART regression tree. Of 1,204 undetermined intent poisoning deaths, CART classified 903 as suicides and 301 as unintentional deaths. The major strength of our study is the use of CART to differentiate with a high degree of accuracy between unintentional and suicidal poisoning deaths among Maryland undetermined intent poisoning deaths.

Changes in ciprofloxacin utilization as shown in a large pharmacy claims database: effects of proximity to criminal anthrax exposure in October 2001.

OBJECTIVE: Identify, during the period of criminal anthrax exposures in October 2001, changes in utilization of ciprofloxacin and geographic patterns of any identified variations. DESIGN: Observational. SETTING: United States. PATIENTS: Individuals making prescription claims through a pharmacy benefits management company. INTERVENTIONS: Analysis of AdvancePCS pharmacy claims database. MAIN OUTCOME MEASURES: Percentage change in ciprofloxacin utilization for 2000 and 2001 and, by locale, for September and October 2001. RESULTS: Utilization of ciprofloxacin tablets was significantly lower in calendar year 2001 than in calendar year 2000 (median decline, 10.3%) for all months except October, when utilization of ciprofloxacin increased 9.8%. During the period of anthrax exposures (October 2001 versus September 2001), affected geographic areas, including New York (an increase of 62.5%), some other Mid-Atlantic states, and Florida (28.5%), had some of the highest percentage increases in the rate of ciprofloxacin utilization. CONCLUSION: Many Americans actively sought prophylaxis with ciprofloxacin during the course of the October 2001 anthrax attack and that utilization was higher in, but not limited to, locales with publicized cases of disease. Pharmacists, clinicians, and public health officials should note that such behavior may be expected in the event of a similar attack and should be familiar with current recommendations for the assessment and management of anthrax exposure.

Tardive dyskinesia risks and metoclopramide use before and after U.S. market withdrawal of cisapride.

OBJECTIVE: To assess risk factors for tardive dyskinesia (TD) in spontaneous reports of metoclopramide and TD and evaluate metoclopramide prescribing patterns before and after withdrawal of cisapride from the market in the United States. DESIGN: Retrospective and observational analyses. SETTING: International metoclopramide adverse event reports and domestic drug-use data for the continental United States. PATIENTS: Users of metoclopramide for 30 days or more who experienced adverse events reported as TD. INTERVENTIONS: Analyses of the Food and Drug Administration Adverse Event Reporting System (AERS) and IMS HEALTH data. MAIN OUTCOME MEASURES: Pharmacoepidemiological patterns in AERS reports and utilization data from IMS HEALTH. RESULTS: The case series comprised 87 reports of primarily older (mean+/-SD, 60+/-22 years ) women (67% of all cases). While average metoclopramide daily dose (33+/-14 mg) was within recommended product labeling limits, duration of use was considerably longer (753+/-951 days). Overall, 37% of the reports included concomitant drugs believed to be TD risk factors. Similarly, 18% of the reports noted comorbid diseases that are considered risk factors for development of TD. Metoclopramide utilization decreased following cisapride marketing in 1993 and increased following cisapride withdrawal in 2000. The majority (62%) of metoclopramide prescriptions were intended for women. Intended use overall increased with age and was highest in the seventh and eighth decades, with nearly one quarter of all utilization being in persons older than 70 years. CONCLUSION: Well-described TD risk factors were common in metoclopramide-associated TD reports. Given the cisapride market withdrawal and associated increased metoclopramide utilization, the incidence of TD may increase accordingly. TD risk factors relative to the intended benefit and duration of use should be considered in metoclopramide prescribing.