A general framework for developing computable clinical phenotype algorithms.

OBJECTIVE: Present a general framework providing high-level guidance to developers of computable algorithms for identifying patients with specific clinical conditions (phenotypes) through a variety of approaches, including but not limited to machine learning and natural language processing methods to incorporate rich electronic health record data. MATERIALS/METHODS: Drawing on extensive prior phenotyping experiences and insights derived from three algorithm development projects conducted specifically for this purpose, our team with expertise in clinical medicine, statistics, informatics, pharmacoepidemiology, and healthcare data science methods conceptualized stages of development and corresponding sets of principles, strategies, and practical guidelines for improving the algorithm development process. RESULTS: We propose five stages of algorithm development and corresponding principles, strategies, and guidelines: 1) assessing fitness-for-purpose, 2) creating gold standard data, 3) feature engineering, 4) model development, and 5) model evaluation. DISCUSSION/CONCLUSION: This framework is intended to provide practical guidance and serve as a basis for future elaboration and extension.

Telegenetics to provide comprehensive prenatal diagnosis.

Telehealth is an effective way to increase access to genetic services and can address several challenges, including geographic barriers, a shortage of interpreter services, and workforce issues, especially for prenatal diagnosis. The addition of prenatal telegenetics to current workflows shows promise in enhancing the delivery of genetic counseling and testing in prenatal care, providing accessibility, accuracy, patient satisfaction, and cost-effectiveness. Further research is needed to explore long-term patient outcomes and the evolving role of telehealth for prenatal diagnosis. Future studies should address the accuracy of diagnoses, the impact of receiving a diagnosis in a virtual setting, and patient outcomes in order to make informed decisions about the appropriate use of telemedicine in prenatal genetics service delivery.

Results From a Prospective, Clinical Study (US-nPower) Evaluating a Miniature Spinal Cord Stimulator for the Management of Chronic, Intractable Pain.

BACKGROUND: Chronic, intractable, neuropathic pain is readily treatable with spinal cord stimulation (SCS). Technological advancements, including device miniaturization, are advancing the field of neuromodulation. OBJECTIVES: We report here the results of an SCS clinical trial to treat chronic, low back and leg pain, with a micro-implantable pulse generator (micro-IPG). STUDY DESIGN: This was a single-arm, prospective, multicenter, postmarket, observational study. SETTING: Patients were recruited from 15 US-based comprehensive pain centers. METHODS: This open-label clinical trial was designed to evaluate the performance of the Nalu™ Neurostimulation System (Nalu Medical, Inc., Carlsbad, CA) in the treatment of low back and leg pain. Patients, who provided informed consent and were successfully screened for study entry, were implanted with temporary trial leads. Patients went on to receive a permanent implant of the leads and micro-IPG if they demonstrated a >= 50% reduction in pain during the temporary trial period. Patient-reported outcomes (PROs), such as pain scores, functional disability, mood, patient impression of change, comfort, therapy use profile, and device ease of use, were captured. RESULTS: At baseline, the average pain Visual Analog Scale (VAS) score was 72.1 ± 17.9 in the leg and 78.0 ± 15.4 in the low back. At 90 days following permanent implant (end of study), pain scores improved by 76% (VAS 18.5 ± 18.8) in the leg and 75% (VAS 19.7 ± 20.8) in the low back. Eighty-six percent  of both leg pain and low back pain patients demonstrated a >= 50% reduction in pain at 90 days following implant. The comfort of the external wearable (Therapy Disc and Adhesive Clip) was rated 1.16 ± 1.53, on average, at 90 days on an 11-point rating scale (0 = very comfortable, 10 = very uncomfortable). All PROs demonstrated statistically significant symptomatic improvement at 90 days following implant of the micro-IPG. LIMITATIONS:   Limitations of this study include the lack of long-term results (beyond 90 days) and a relatively small sample size of 35 patients who were part of the analysis; additionally, there was no control arm or randomization as this was a single-arm study, without a comparator, designed to document the efficacy and safety of the device. Therefore, no direct comparisons to other SCS systems were possible. CONCLUSIONS: This clinical study demonstrated profound leg and low back pain relief in terms of overall pain reduction, as well as the proportion of therapy responders. The study patients reported the wearable aspects of the system to be very comfortable.

Six Years of the US Food and Drug Administration's Postmarket Active Risk Identification and Analysis System in the Sentinel Initiative: Implications for Real World Evidence Generation.

Congress mandated the creation of a postmarket Active Risk Identification and Analysis (ARIA) system containing data on 100 million individuals for monitoring risks associated with drug and biologic products using data from disparate sources to complement the US Food and Drug Administration's (FDA's) existing postmarket capabilities. We report on the first 6 years of ARIA utilization in the Sentinel System (2016-2021). The FDA has used the ARIA system to evaluate 133 safety concerns; 54 of these evaluations have closed with regulatory determinations, whereas the rest remain in progress. If the ARIA system and the FDA's Adverse Event Reporting System are deemed insufficient to address a safety concern, then the FDA may issue a postmarket requirement to a product's manufacturer. One hundred ninety-seven ARIA insufficiency determinations have been made. The most common situation for which ARIA was found to be insufficient is the evaluation of adverse pregnancy and fetal outcomes following in utero drug exposure, followed by neoplasms and death. ARIA was most likely to be sufficient for thromboembolic events, which have high positive predictive value in claims data alone and do not require supplemental clinical data. The lessons learned from this experience illustrate the continued challenges using administrative claims data, especially to define novel clinical outcomes. This analysis can help to identify where more granular clinical data are needed to fill gaps to improve the use of real-world data for drug safety analyses and provide insights into what is needed to efficiently generate high-quality real-world evidence for efficacy.

Trends and opportunities in computable clinical phenotyping: A scoping review.

Identifying patient cohorts meeting the criteria of specific phenotypes is essential in biomedicine and particularly timely in precision medicine. Many research groups deliver pipelines that automatically retrieve and analyze data elements from one or more sources to automate this task and deliver high-performing computable phenotypes. We applied a systematic approach based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines to conduct a thorough scoping review on computable clinical phenotyping. Five databases were searched using a query that combined the concepts of automation, clinical context, and phenotyping. Subsequently, four reviewers screened 7960 records (after removing over 4000 duplicates) and selected 139 that satisfied the inclusion criteria. This dataset was analyzed to extract information on target use cases, data-related topics, phenotyping methodologies, evaluation strategies, and portability of developed solutions. Most studies supported patient cohort selection without discussing the application to specific use cases, such as precision medicine. Electronic Health Records were the primary source in 87.1 % (N = 121) of all studies, and International Classification of Diseases codes were heavily used in 55.4 % (N = 77) of all studies, however, only 25.9 % (N = 36) of the records described compliance with a common data model. In terms of the presented methods, traditional Machine Learning (ML) was the dominant method, often combined with natural language processing and other approaches, while external validation and portability of computable phenotypes were pursued in many cases. These findings revealed that defining target use cases precisely, moving away from sole ML strategies, and evaluating the proposed solutions in the real setting are essential opportunities for future work. There is also momentum and an emerging need for computable phenotyping to support clinical and epidemiological research and precision medicine.

Improving Methods of Identifying Anaphylaxis for Medical Product Safety Surveillance Using Natural Language Processing and Machine Learning.

We sought to determine whether machine learning and natural language processing (NLP) applied to electronic medical records could improve performance of automated health-care claims-based algorithms to identify anaphylaxis events using data on 516 patients with outpatient, emergency department, or inpatient anaphylaxis diagnosis codes during 2015-2019 in 2 integrated health-care institutions in the Northwest United States. We used one site's manually reviewed gold-standard outcomes data for model development and the other's for external validation based on cross-validated area under the receiver operating characteristic curve (AUC), positive predictive value (PPV), and sensitivity. In the development site 154 (64%) of 239 potential events met adjudication criteria for anaphylaxis compared with 180 (65%) of 277 in the validation site. Logistic regression models using only structured claims data achieved a cross-validated AUC of 0.58 (95% CI: 0.54, 0.63). Machine learning improved cross-validated AUC to 0.62 (0.58, 0.66); incorporating NLP-derived covariates further increased cross-validated AUCs to 0.70 (0.66, 0.75) in development and 0.67 (0.63, 0.71) in external validation data. A classification threshold with cross-validated PPV of 79% and cross-validated sensitivity of 66% in development data had cross-validated PPV of 78% and cross-validated sensitivity of 56% in external data. Machine learning and NLP-derived data improved identification of validated anaphylaxis events.

Validation of Acute Pancreatitis Among Adults in an Integrated Healthcare System.

BACKGROUND: Acute pancreatitis is a serious gastrointestinal disease that is an important target for drug safety surveillance. Little is known about the accuracy of ICD-10 codes for acute pancreatitis in the United States, or their performance in specific clinical settings. We conducted a validation study to assess the accuracy of acute pancreatitis ICD-10 diagnosis codes in inpatient, emergency department (ED), and outpatient settings. METHODS: We reviewed electronic medical records for encounters with acute pancreatitis diagnosis codes in an integrated healthcare system from October 2015 to December 2019. Trained abstractors and physician adjudicators determined whether events met criteria for acute pancreatitis. RESULTS: Out of 1,844 eligible events, we randomly sampled 300 for review. Across all clinical settings, 182 events met validation criteria for an overall positive predictive value (PPV) of 61% (95% confidence intervals [CI] = 55, 66). The PPV was 87% (95% CI = 79, 92%) for inpatient codes, but only 45% for ED (95% CI = 35, 54%) and outpatient (95% CI = 34, 55%) codes. ED and outpatient encounters accounted for 43% of validated events. Acute pancreatitis codes from any encounter type with lipase >3 times the upper limit of normal had a PPV of 92% (95% CI = 86, 95%) and identified 85% of validated events (95% CI = 79, 89%), while codes with lipase <3 times the upper limit of normal had a PPV of only 22% (95% CI = 16, 30%). CONCLUSIONS: These results suggest that ICD-10 codes accurately identified acute pancreatitis in the inpatient setting, but not in the ED and outpatient settings. Laboratory data substantially improved algorithm performance.

The US Food and Drug Administration Sentinel System: a national resource for a learning health system.

The US Food and Drug Administration (FDA) created the Sentinel System in response to a requirement in the FDA Amendments Act of 2007 that the agency establish a system for monitoring risks associated with drug and biologic products using data from disparate sources. The Sentinel System has completed hundreds of analyses, including many that have directly informed regulatory decisions. The Sentinel System also was designed to support a national infrastructure for a learning health system. Sentinel governance and guiding principles were designed to facilitate Sentinel's role as a national resource. The Sentinel System infrastructure now supports multiple non-FDA projects for stakeholders ranging from regulated industry to other federal agencies, international regulators, and academics. The Sentinel System is a working example of a learning health system that is expanding with the potential to create a global learning health system that can support medical product safety assessments and other research.

On the use of the likelihood ratio test methodology in pharmacovigilance.

The safety of medical products due to adverse events (AE) from drugs, therapeutic biologics, and medical devices is a major public health concern worldwide. Likelihood ratio test (LRT) approaches to pharmacovigilance constitute a class of rigorous statistical tools that permit objective identification of AEs of a specific drug and/or a class of drugs cataloged in spontaneous reporting system databases. However, the existing LRT approaches encounter certain theoretical and computational challenges when an underlying Poisson model assumption is violated, including in cases of zero-inflated data. We briefly review existing LRT approaches and propose a novel class of (pseudo-) LRT methods to address these challenges. Our approach uses an alternative parametrization to formulate a unified framework with a common test statistic that can handle both Poisson and zero-inflated Poisson (ZIP) models. The proposed framework is computationally efficient, and it reveals deeper insights into the comparative behaviors of the Poisson and the ZIP models for handling AE data. Our extensive simulation studies document notably superior performances of the proposed methods over existing approaches particularly under zero-inflation, both in terms of statistical (eg, much better control of the nominal level and false discovery rate with substantially enhanced power) and computational ( ∼ $$ \sim $$ 100-500-fold gains in average running times) performance metrics. An application of our method on the statin drug class from the FDA FAERS database reveals interesting insights on potential AEs. An R package, pvLRT, implementing our methods has been released in the public domain.

Impact of Telemedicine on Prenatal Counseling at a Tertiary Fetal Center: A Mixed Methods Study.

INTRODUCTION: COVID-19 spurred an unprecedented transition from in-person to telemedicine visits in March 2020 at our institution for all prenatal counseling sessions. This study aims to explore differences in demographics of expectant mothers evaluated pre- and post-telemedicine implementation and to explore the patient experience with telemedicine. METHODS: A mixed methods study was completed for mothers with a pregnancy complicated by a fetal surgical anomaly who visited a large tertiary fetal center. Using medical records as quantitative data, patient information was collected for all prenatal visits from 3/2019 to 3/2021. The sample was grouped into pre- and post-telemedicine implementation (based on transition date of 3/2020). Univariate analysis was used to compare demographics between the study groups. Statistical significance was defined as P < 0.05. Eighteen semi-structured interviews were conducted from 8/2021 to 12/2021 to explore patients' experiences. Line-by-line coding and thematic analysis was performed to develop emerging themes. RESULTS: 292 pregnancies were evaluated from 3/2019 to 3/2021 (pre-telemedicine 123, post-telemedicine 169). There was no significant difference in self-reported race (P = 0.28), ethnicity (P = 0.46), or primary language (P = 0.98). In qualitative interviews, patients reported advantages to telemedicine, including the convenience of the modality with the option to conduct their session in familiar settings (e.g., home) and avoid stressors (e.g., travel to the medical center and finding childcare). Some women reported difficulties establishing a physician-patient connection and a preference for in-person consultations. CONCLUSIONS: There was no difference in patient demographics at our fetal center in the year leading up to, and the time following, a significant transition to telemedicine. However, patients had unique perspectives on the advantages and disadvantages of the telemedicine experience. To ensure patient centered care, these findings suggest patient preference should be considered when scheduling outpatient surgical counseling and visits.

"Artificial Intelligence" for Pharmacovigilance: Ready for Prime Time?

There is great interest in the application of 'artificial intelligence' (AI) to pharmacovigilance (PV). Although US FDA is broadly exploring the use of AI for PV, we focus on the application of AI to the processing and evaluation of Individual Case Safety Reports (ICSRs) submitted to the FDA Adverse Event Reporting System (FAERS). We describe a general framework for considering the readiness of AI for PV, followed by some examples of the application of AI to ICSR processing and evaluation in industry and FDA. We conclude that AI can usefully be applied to some aspects of ICSR processing and evaluation, but the performance of current AI algorithms requires a 'human-in-the-loop' to ensure good quality. We identify outstanding scientific and policy issues to be addressed before the full potential of AI can be exploited for ICSR processing and evaluation, including approaches to quality assurance of 'human-in-the-loop' AI systems, large-scale, publicly available training datasets, a well-defined and computable 'cognitive framework', a formal sociotechnical framework for applying AI to PV, and development of best practices for applying AI to PV. Practical experience with stepwise implementation of AI for ICSR processing and evaluation will likely provide important lessons that will inform the necessary policy and regulatory framework to facilitate widespread adoption and provide a foundation for further development of AI approaches to other aspects of PV.

Emerging technologies and their impact on regulatory science.

There is an evolution and increasing need for the utilization of emerging cellular, molecular and in silico technologies and novel approaches for safety assessment of food, drugs, and personal care products. Convergence of these emerging technologies is also enabling rapid advances and approaches that may impact regulatory decisions and approvals. Although the development of emerging technologies may allow rapid advances in regulatory decision making, there is concern that these new technologies have not been thoroughly evaluated to determine if they are ready for regulatory application, singularly or in combinations. The magnitude of these combined technical advances may outpace the ability to assess fit for purpose and to allow routine application of these new methods for regulatory purposes. There is a need to develop strategies to evaluate the new technologies to determine which ones are ready for regulatory use. The opportunity to apply these potentially faster, more accurate, and cost-effective approaches remains an important goal to facilitate their incorporation into regulatory use. However, without a clear strategy to evaluate emerging technologies rapidly and appropriately, the value of these efforts may go unrecognized or may take longer. It is important for the regulatory science field to keep up with the research in these technically advanced areas and to understand the science behind these new approaches. The regulatory field must understand the critical quality attributes of these novel approaches and learn from each other's experience so that workforces can be trained to prepare for emerging global regulatory challenges. Moreover, it is essential that the regulatory community must work with the technology developers to harness collective capabilities towards developing a strategy for evaluation of these new and novel assessment tools.

Broadening the reach of the FDA Sentinel system: A roadmap for integrating electronic health record data in a causal analysis framework.

The Sentinel System is a major component of the United States Food and Drug Administration's (FDA) approach to active medical product safety surveillance. While Sentinel has historically relied on large quantities of health insurance claims data, leveraging longitudinal electronic health records (EHRs) that contain more detailed clinical information, as structured and unstructured features, may address some of the current gaps in capabilities. We identify key challenges when using EHR data to investigate medical product safety in a scalable and accelerated way, outline potential solutions, and describe the Sentinel Innovation Center's initiatives to put solutions into practice by expanding and strengthening the existing system with a query-ready, large-scale data infrastructure of linked EHR and claims data. We describe our initiatives in four strategic priority areas: (1) data infrastructure, (2) feature engineering, (3) causal inference, and (4) detection analytics, with the goal of incorporating emerging data science innovations to maximize the utility of EHR data for medical product safety surveillance.

Fatal Co-infections with SARS-CoV-2 and Legionella pneumophila, England.

Both Legionella pneumophila and severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) can cause pneumonia. L. pneumophila is acquired from water sources, sometimes in healthcare settings. We report 2 fatal cases of L. pneumophila and SARS-CoV-2 co-infection in England. Clinicians should be aware of possible L. pneumophila infections among SARS-CoV-2 patients.

Feature engineering and machine learning for causality assessment in pharmacovigilance: Lessons learned from application to the FDA Adverse Event Reporting System.

BACKGROUND: Our objective was to support the automated classification of Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS) reports for their usefulness in assessing the possibility of a causal relationship between a drug product and an adverse event. METHOD: We used a data set of 326 redacted FAERS reports that was previously annotated using a modified version of the World Health Organization-Uppsala Monitoring Centre criteria for drug causality assessment by a group of SEs at the FDA and supported a similar study on the classification of reports using supervised machine learning and text engineering methods. We explored many potential features, including the incorporation of natural language processing on report text and information from external data sources, for supervised learning and developed models for predicting the classification status of reports. We then evaluated the models on a larger data set of previously unseen reports. RESULTS: The best-performing models achieved recall and F1 scores on both data sets above 0.80 for the identification of assessable reports (i.e. those containing enough information to make an informed causality assessment) and above 0.75 for the identification of reports meeting at least a Possible causality threshold. CONCLUSIONS: Causal inference from FAERS reports depends on many components with complex logical relationships that are yet to be made fully computable. Efforts focused on readily addressable tasks, such as quickly eliminating unassessable reports, fit naturally in SE's thought processes to provide real enhancements for FDA workflows.

A Series of Tattoo-associated Mycobacterium Fortuitum Infections.

Mycobacterium fortuitum is a rapidly growing mycobacterium known to spread through many sources, including tap water. This organism can have variable presentation between patients which can lead to a delay in diagnosis. Here, we report a series of eight cases of tattoo-associated M. fortuitum infections that presented between December 2010 and January 2011, which were later linked to a single tattoo provider using gray tattoo ink made by diluting black ink with nonsterile tap water. In this case series, we emphasize the lack of pathognomonic features of these infections, the variability in culture and biopsy results, the importance of obtaining a culture in addition to a biopsy, and the importance of identifying the source of infection when determining management.

Gibbon strategies in a food competition task.

Social primates face conflicts of interest with other partners when their individual and collective interests collide. Despite living in small, primarily bonded, groups compared to other social primates, gibbons are not exempt from these conflicts in their everyday lives. In the current task, we asked whether dyads of gibbons would solve a conflict of interest over food rewards. We presented dyads of gibbons with a situation in which they could decide whether to take an active role and pull a handle to release food rewards at a distance or take a passive role and avoid action. In this situation, the passive partner could take an advantageous position to obtain the rewards over the active partner. Gibbons participated in three conditions: a control condition with no food rewards, a test condition with indirect food rewards and a test condition with direct food rewards. In both test conditions, five rewards were released at a distance from the handle. In addition, the active individual could obtain one extra food reward from the handle in the direct food condition. We found that gibbons acted more often in the two conditions involving food rewards, and waited longer in the indirect compared to the direct food condition, thus suggesting that they understood the task contingencies. Surprisingly, we found that in a majority of dyads, individuals in the active role obtained most of the payoff compared to individuals in the passive role in both food conditions. Furthermore, in some occasions individuals in the active role did not approach the location where the food was released. These results suggest that while gibbons may strategize to maximize benefits in a competitive food task, they often allowed their partners to obtain better rewards. Our results highlight the importance of social tolerance and motivation as drivers promoting cooperation in these species.