A simulation-based bias analysis to assess the impact of unmeasured confounding when designing non-randomized database studies.

BACKGROUND: Unmeasured confounding is often raised as a source of potential bias during the design of non-randomized studies but quantifying such concerns is challenging. METHODS: We developed a simulation-based approach to assess the potential impact of unmeasured confounding during the study design stage. The approach involved generation of hypothetical individual-level cohorts using realistic parameters including a binary treatment (prevalence 25%), a time-to-event outcome (incidence 5%), 13 measured covariates, a binary unmeasured confounder (u1, 10%), and a binary measured 'proxy' variable (p1) correlated with u1. Strength of unmeasured confounding and correlations between u1 and p1 were varied in simulation scenarios. Treatment effects were estimated with, a) no adjustment, b) adjustment for measured confounders (Level 1), c) adjustment for measured confounders and their proxy (Level 2). We computed absolute standardized mean differences in u1 and p1 and relative bias with each level of adjustment. RESULTS: Across all scenarios, Level 2 adjustment led to improvement in balance of u1, but this improvement was highly dependent on the correlation between u1 and p1. Level 2 adjustments also had lower relative bias than Level 1 adjustments (in strong u1 scenarios: relative bias of 9.2%, 12.2%, 13.5% at correlations 0.7, 0.5, and 0.3, respectively versus 16.4%, 15.8%, 15.0% for Level 1, respectively). CONCLUSION: An approach using simulated individual-level data was useful to explicitly convey the potential for bias due to unmeasured confounding while designing non-randomized studies and can be helpful in informing design choices.

Telegenetics to provide comprehensive prenatal diagnosis.

Telehealth is an effective way to increase access to genetic services and can address several challenges, including geographic barriers, a shortage of interpreter services, and workforce issues, especially for prenatal diagnosis. The addition of prenatal telegenetics to current workflows shows promise in enhancing the delivery of genetic counseling and testing in prenatal care, providing accessibility, accuracy, patient satisfaction, and cost-effectiveness. Further research is needed to explore long-term patient outcomes and the evolving role of telehealth for prenatal diagnosis. Future studies should address the accuracy of diagnoses, the impact of receiving a diagnosis in a virtual setting, and patient outcomes in order to make informed decisions about the appropriate use of telemedicine in prenatal genetics service delivery.

Trust but Verify: Lessons Learned for the Application of AI to Case-Based Clinical Decision-Making From Postmarketing Drug Safety Assessment at the US Food and Drug Administration.

Adverse drug reactions are a common cause of morbidity in health care. The US Food and Drug Administration (FDA) evaluates individual case safety reports of adverse events (AEs) after submission to the FDA Adverse Event Reporting System as part of its surveillance activities. Over the past decade, the FDA has explored the application of artificial intelligence (AI) to evaluate these reports to improve the efficiency and scientific rigor of the process. However, a gap remains between AI algorithm development and deployment. This viewpoint aims to describe the lessons learned from our experience and research needed to address both general issues in case-based reasoning using AI and specific needs for individual case safety report assessment. Beginning with the recognition that the trustworthiness of the AI algorithm is the main determinant of its acceptance by human experts, we apply the Diffusion of Innovations theory to help explain why certain algorithms for evaluating AEs at the FDA were accepted by safety reviewers and others were not. This analysis reveals that the process by which clinicians decide from case reports whether a drug is likely to cause an AE is not well defined beyond general principles. This makes the development of high performing, transparent, and explainable AI algorithms challenging, leading to a lack of trust by the safety reviewers. Even accounting for the introduction of large language models, the pharmacovigilance community needs an improved understanding of causal inference and of the cognitive framework for determining the causal relationship between a drug and an AE. We describe specific future research directions that underpin facilitating implementation and trust in AI for drug safety applications, including improved methods for measuring and controlling of algorithmic uncertainty, computational reproducibility, and clear articulation of a cognitive framework for causal inference in case-based reasoning.

Improving Methods of Identifying Anaphylaxis for Medical Product Safety Surveillance Using Natural Language Processing and Machine Learning.

We sought to determine whether machine learning and natural language processing (NLP) applied to electronic medical records could improve performance of automated health-care claims-based algorithms to identify anaphylaxis events using data on 516 patients with outpatient, emergency department, or inpatient anaphylaxis diagnosis codes during 2015-2019 in 2 integrated health-care institutions in the Northwest United States. We used one site's manually reviewed gold-standard outcomes data for model development and the other's for external validation based on cross-validated area under the receiver operating characteristic curve (AUC), positive predictive value (PPV), and sensitivity. In the development site 154 (64%) of 239 potential events met adjudication criteria for anaphylaxis compared with 180 (65%) of 277 in the validation site. Logistic regression models using only structured claims data achieved a cross-validated AUC of 0.58 (95% CI: 0.54, 0.63). Machine learning improved cross-validated AUC to 0.62 (0.58, 0.66); incorporating NLP-derived covariates further increased cross-validated AUCs to 0.70 (0.66, 0.75) in development and 0.67 (0.63, 0.71) in external validation data. A classification threshold with cross-validated PPV of 79% and cross-validated sensitivity of 66% in development data had cross-validated PPV of 78% and cross-validated sensitivity of 56% in external data. Machine learning and NLP-derived data improved identification of validated anaphylaxis events.

Validation of Acute Pancreatitis Among Adults in an Integrated Healthcare System.

BACKGROUND: Acute pancreatitis is a serious gastrointestinal disease that is an important target for drug safety surveillance. Little is known about the accuracy of ICD-10 codes for acute pancreatitis in the United States, or their performance in specific clinical settings. We conducted a validation study to assess the accuracy of acute pancreatitis ICD-10 diagnosis codes in inpatient, emergency department (ED), and outpatient settings. METHODS: We reviewed electronic medical records for encounters with acute pancreatitis diagnosis codes in an integrated healthcare system from October 2015 to December 2019. Trained abstractors and physician adjudicators determined whether events met criteria for acute pancreatitis. RESULTS: Out of 1,844 eligible events, we randomly sampled 300 for review. Across all clinical settings, 182 events met validation criteria for an overall positive predictive value (PPV) of 61% (95% confidence intervals [CI] = 55, 66). The PPV was 87% (95% CI = 79, 92%) for inpatient codes, but only 45% for ED (95% CI = 35, 54%) and outpatient (95% CI = 34, 55%) codes. ED and outpatient encounters accounted for 43% of validated events. Acute pancreatitis codes from any encounter type with lipase >3 times the upper limit of normal had a PPV of 92% (95% CI = 86, 95%) and identified 85% of validated events (95% CI = 79, 89%), while codes with lipase <3 times the upper limit of normal had a PPV of only 22% (95% CI = 16, 30%). CONCLUSIONS: These results suggest that ICD-10 codes accurately identified acute pancreatitis in the inpatient setting, but not in the ED and outpatient settings. Laboratory data substantially improved algorithm performance.

Trends and opportunities in computable clinical phenotyping: A scoping review.

Identifying patient cohorts meeting the criteria of specific phenotypes is essential in biomedicine and particularly timely in precision medicine. Many research groups deliver pipelines that automatically retrieve and analyze data elements from one or more sources to automate this task and deliver high-performing computable phenotypes. We applied a systematic approach based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines to conduct a thorough scoping review on computable clinical phenotyping. Five databases were searched using a query that combined the concepts of automation, clinical context, and phenotyping. Subsequently, four reviewers screened 7960 records (after removing over 4000 duplicates) and selected 139 that satisfied the inclusion criteria. This dataset was analyzed to extract information on target use cases, data-related topics, phenotyping methodologies, evaluation strategies, and portability of developed solutions. Most studies supported patient cohort selection without discussing the application to specific use cases, such as precision medicine. Electronic Health Records were the primary source in 87.1 % (N = 121) of all studies, and International Classification of Diseases codes were heavily used in 55.4 % (N = 77) of all studies, however, only 25.9 % (N = 36) of the records described compliance with a common data model. In terms of the presented methods, traditional Machine Learning (ML) was the dominant method, often combined with natural language processing and other approaches, while external validation and portability of computable phenotypes were pursued in many cases. These findings revealed that defining target use cases precisely, moving away from sole ML strategies, and evaluating the proposed solutions in the real setting are essential opportunities for future work. There is also momentum and an emerging need for computable phenotyping to support clinical and epidemiological research and precision medicine.

On the use of the likelihood ratio test methodology in pharmacovigilance.

The safety of medical products due to adverse events (AE) from drugs, therapeutic biologics, and medical devices is a major public health concern worldwide. Likelihood ratio test (LRT) approaches to pharmacovigilance constitute a class of rigorous statistical tools that permit objective identification of AEs of a specific drug and/or a class of drugs cataloged in spontaneous reporting system databases. However, the existing LRT approaches encounter certain theoretical and computational challenges when an underlying Poisson model assumption is violated, including in cases of zero-inflated data. We briefly review existing LRT approaches and propose a novel class of (pseudo-) LRT methods to address these challenges. Our approach uses an alternative parametrization to formulate a unified framework with a common test statistic that can handle both Poisson and zero-inflated Poisson (ZIP) models. The proposed framework is computationally efficient, and it reveals deeper insights into the comparative behaviors of the Poisson and the ZIP models for handling AE data. Our extensive simulation studies document notably superior performances of the proposed methods over existing approaches particularly under zero-inflation, both in terms of statistical (eg, much better control of the nominal level and false discovery rate with substantially enhanced power) and computational ( ∼ $$ \sim $$ 100-500-fold gains in average running times) performance metrics. An application of our method on the statin drug class from the FDA FAERS database reveals interesting insights on potential AEs. An R package, pvLRT, implementing our methods has been released in the public domain.

Impact of Telemedicine on Prenatal Counseling at a Tertiary Fetal Center: A Mixed Methods Study.

INTRODUCTION: COVID-19 spurred an unprecedented transition from in-person to telemedicine visits in March 2020 at our institution for all prenatal counseling sessions. This study aims to explore differences in demographics of expectant mothers evaluated pre- and post-telemedicine implementation and to explore the patient experience with telemedicine. METHODS: A mixed methods study was completed for mothers with a pregnancy complicated by a fetal surgical anomaly who visited a large tertiary fetal center. Using medical records as quantitative data, patient information was collected for all prenatal visits from 3/2019 to 3/2021. The sample was grouped into pre- and post-telemedicine implementation (based on transition date of 3/2020). Univariate analysis was used to compare demographics between the study groups. Statistical significance was defined as P < 0.05. Eighteen semi-structured interviews were conducted from 8/2021 to 12/2021 to explore patients' experiences. Line-by-line coding and thematic analysis was performed to develop emerging themes. RESULTS: 292 pregnancies were evaluated from 3/2019 to 3/2021 (pre-telemedicine 123, post-telemedicine 169). There was no significant difference in self-reported race (P = 0.28), ethnicity (P = 0.46), or primary language (P = 0.98). In qualitative interviews, patients reported advantages to telemedicine, including the convenience of the modality with the option to conduct their session in familiar settings (e.g., home) and avoid stressors (e.g., travel to the medical center and finding childcare). Some women reported difficulties establishing a physician-patient connection and a preference for in-person consultations. CONCLUSIONS: There was no difference in patient demographics at our fetal center in the year leading up to, and the time following, a significant transition to telemedicine. However, patients had unique perspectives on the advantages and disadvantages of the telemedicine experience. To ensure patient centered care, these findings suggest patient preference should be considered when scheduling outpatient surgical counseling and visits.

Fatal Co-infections with SARS-CoV-2 and Legionella pneumophila, England.

Both Legionella pneumophila and severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) can cause pneumonia. L. pneumophila is acquired from water sources, sometimes in healthcare settings. We report 2 fatal cases of L. pneumophila and SARS-CoV-2 co-infection in England. Clinicians should be aware of possible L. pneumophila infections among SARS-CoV-2 patients.

Identification and Validation of Anaphylaxis Using Electronic Health Data in a Population-based Setting.

BACKGROUND: Anaphylaxis is a life-threatening allergic reaction that is difficult to identify accurately with administrative data. We conducted a population-based validation study to assess the accuracy of ICD-10 diagnosis codes for anaphylaxis in outpatient, emergency department, and inpatient settings. METHODS: In an integrated healthcare system in Washington State, we obtained medical records from healthcare encounters with anaphylaxis diagnosis codes (potential events) from October 2015 to December 2018. To capture events missed by anaphylaxis diagnosis codes, we also obtained records on a sample of serious allergic and drug reactions. Two physicians determined whether potential events met established clinical criteria for anaphylaxis (validated events). RESULTS: Out of 239 potential events with anaphylaxis diagnosis codes, the overall positive predictive value (PPV) for validated events was 64% (95% CI = 58 to 70). The PPV decreased with increasing age. Common precipitants for anaphylaxis were food (39%), medications (35%), and insect bite or sting (12%). The sensitivity of emergency department and inpatient anaphylaxis diagnosis codes for all validated events was 58% (95% CI = 51 to 65), but sensitivity increased to 95% (95% CI = 74 to 99) when outpatient diagnosis codes were included. Using information from all validated events and sampling weights, the incidence rate for anaphylaxis was 3.6 events per 10,000 person-years (95% CI = 3.1 to 4.0). CONCLUSIONS: In this population-based setting, ICD-10 diagnosis codes for anaphylaxis from emergency department and inpatient settings had moderate PPV and sensitivity for validated events. These findings have implications for epidemiologic studies that seek to estimate risks of anaphylaxis using electronic health data.

A COVID-19-ready public health surveillance system: The Food and Drug Administration's Sentinel System.

The US Food and Drug Administration's Sentinel System was established in 2009 to use routinely collected electronic health data for improving the national capability to assess post-market medical product safety. Over more than a decade, Sentinel has become an integral part of FDA's surveillance capabilities and has been used to conduct analyses that have contributed to regulatory decisions. FDA's role in the COVID-19 pandemic response has necessitated an expansion and enhancement of Sentinel. Here we describe how the Sentinel System has supported FDA's response to the COVID-19 pandemic. We highlight new capabilities developed, key data generated to date, and lessons learned, particularly with respect to working with inpatient electronic health record data. Early in the pandemic, Sentinel developed a multi-pronged approach to support FDA's anticipated data and analytic needs. It incorporated new data sources, created a rapidly refreshed database, developed protocols to assess the natural history of COVID-19, validated a diagnosis-code based algorithm for identifying patients with COVID-19 in administrative claims data, and coordinated with other national and international initiatives. Sentinel is poised to answer important questions about the natural history of COVID-19 and is positioned to use this information to study the use, safety, and potentially the effectiveness of medical products used for COVID-19 prevention and treatment.

Leveraging the Capabilities of the FDA's Sentinel System To Improve Kidney Care.

The Sentinel System is a national electronic postmarketing resource established by the US Food and Drug Administration to support assessment of the safety and effectiveness of marketed medical products. It has built a large, multi-institutional, distributed data network that contains comprehensive electronic health data, covering about 700 million person-years of longitudinal observation time nationwide. With its sophisticated infrastructure and a large selection of flexible analytic tools, the Sentinel System permits rapid and secure analyses, while preserving patient privacy and health-system autonomy. The Sentinel System also offers enhanced capabilities, including accessing full-text medical records, supporting randomized clinical trials embedded in healthcare delivery systems, and facilitating effective collection of patient-reported data using mobile devices, among many other research programs. The nephrology research community can use the infrastructure, tools, and data that this national resource offers for evidence generation. This review summarizes the Sentinel System and its ability to rapidly generate high-quality, real-world evidence; discusses the program's experience in, and potential for, addressing gaps in kidney care; and outlines avenues for conducting research, leveraging this national resource in collaboration with Sentinel investigators.

An evaluation of statistical approaches to postmarketing surveillance.

Safety of medical products presents a serious concern worldwide. Surveillance systems of postmarket medical products have been established for continual monitoring of adverse events (AEs) in many countries, and the proliferation of electronic health record systems further facilitates continual monitoring for AEs. We review existing statistical methods for signal detection that are mostly in use in postmarketing safety surveillance of spontaneously reported AEs and we study their performance characteristics by simulation. We compare those with the likelihood ratio test (LRT) method (appropriately modified for use in pharmacovigilance) and use three different methods to generate data (AE based, drug based, and a modification of the method of Ahmed et al). Performance metrics include type I error, power, sensitivity, and false discovery rate, among others. The results show superior performance of the LRT method in almost all simulation experiments. An application to the FDA Adverse Event Reporting System database is illustrated using rhabdomyolysis-related preferred terms reported to FDA during the third-quarter of 2014 to the first-quarter of 2017 for statin drugs. We present a critical discussion and recommendations for use of these methods.

Evaluation of Natural Language Processing (NLP) systems to annotate drug product labeling with MedDRA terminology.

INTRODUCTION: The FDA Adverse Event Reporting System (FAERS) is a primary data source for identifying unlabeled adverse events (AEs) in a drug or biologic drug product's postmarketing phase. Many AE reports must be reviewed by drug safety experts to identify unlabeled AEs, even if the reported AEs are previously identified, labeled AEs. Integrating the labeling status of drug product AEs into FAERS could increase report triage and review efficiency. Medical Dictionary for Regulatory Activities (MedDRA) is the standard for coding AE terms in FAERS cases. However, drug manufacturers are not required to use MedDRA to describe AEs in product labels. We hypothesized that natural language processing (NLP) tools could assist in automating the extraction and MedDRA mapping of AE terms in drug product labels. MATERIALS AND METHODS: We evaluated the performance of three NLP systems, (ETHER, I2E, MetaMap) for their ability to extract AE terms from drug labels and translate the terms to MedDRA Preferred Terms (PTs). Pharmacovigilance-based annotation guidelines for extracting AE terms from drug labels were developed for this study. We compared each system's output to MedDRA PT AE lists, manually mapped by FDA pharmacovigilance experts using the guidelines, for ten drug product labels known as the "gold standard AE list" (GSL) dataset. Strict time and configuration conditions were imposed in order to test each system's capabilities under conditions of no human intervention and minimal system configuration. Each NLP system's output was evaluated for precision, recall and F measure in comparison to the GSL. A qualitative error analysis (QEA) was conducted to categorize a random sample of each NLP system's false positive and false negative errors. RESULTS: A total of 417, 278, and 250 false positive errors occurred in the ETHER, I2E, and MetaMap outputs, respectively. A total of 100, 80, and 187 false negative errors occurred in ETHER, I2E, and MetaMap outputs, respectively. Precision ranged from 64% to 77%, recall from 64% to 83% and F measure from 67% to 79%. I2E had the highest precision (77%), recall (83%) and F measure (79%). ETHER had the lowest precision (64%). MetaMap had the lowest recall (64%). The QEA found that the most prevalent false positive errors were context errors such as "Context error/General term", "Context error/Instructions or monitoring parameters", "Context error/Medical history preexisting condition underlying condition risk factor or contraindication", and "Context error/AE manifestations or secondary complication". The most prevalent false negative errors were in the "Incomplete or missed extraction" error category. Missing AE terms were typically due to long terms, or terms containing non-contiguous words which do not correspond exactly to MedDRA synonyms. MedDRA mapping errors were a minority of errors for ETHER and I2E but were the most prevalent false positive errors for MetaMap. CONCLUSIONS: The results demonstrate that it may be feasible to use NLP tools to extract and map AE terms to MedDRA PTs. However, the NLP tools we tested would need to be modified or reconfigured to lower the error rates to support their use in a regulatory setting. Tools specific for extracting AE terms from drug labels and mapping the terms to MedDRA PTs may need to be developed to support pharmacovigilance. Conducting research using additional NLP systems on a larger, diverse GSL would also be informative.

Evaluating automated approaches to anaphylaxis case classification using unstructured data from the FDA Sentinel System.

INTRODUCTION: In May 2008, the Food and Drug Administration launched the Sentinel Initiative, a multi-year program for the establishment of a national electronic monitoring system for medical product safety that led, in 2016, to the launch of the full Sentinel System. Under the Mini-Sentinel pilot, several algorithms for identifying health outcomes of interest, including one for anaphylaxis, were developed and evaluated using data available from the Sentinel common data model. PURPOSE: To evaluate whether features extracted from unstructured narrative data using natural language processing (NLP) could be used to classify anaphylaxis cases. METHODS: Using previously developed methods, we extracted features from unstructured narrative data using NLP and applied rule-based and similarity-based algorithms to identify anaphylaxis among 62 potential cases previously classified by human experts as anaphylaxis (N = 33), not anaphylaxis (N = 27), and unknown (N = 2). RESULTS: The rule-based and similarity-based approaches demonstrated almost equal performance (recall 100% vs 100%, precision 60.3% vs 57.4%, F-measure: 0.753 vs 0.729). Reasons for misclassification included the inability of the algorithms to make the same clinical judgments as human experts about the timing, severity, or presence of alternative explanations; and the identification of terms consistent with anaphylaxis but present in conditions other than anaphylaxis. CONCLUSIONS: Although precision needs to be improved before these algorithms could be used without human review, we demonstrated that applying rule-based and similarity-based algorithms to unstructured narrative information from clinical records can be used for classification of anaphylaxis in the Sentinel System. Further development and assessment of these methods in the Sentinel System are warranted.

Analysis of USA Powerlifting Federation Data From January 1, 2012-June 11, 2016.

Ball, R and Weidman, D. Analysis of USA Powerlifting federation data from January 1, 2012-June 11, 2016. J Strength Cond Res 32(7): 1843-1851, 2018-In this article, we report 47,913 officially judged contestant results from powerlifting matches from January 1, 2012 to June 11, 2016 for the USA Powerlifting Federation. We found age and sex to be the most complex factors in predicting powerlifting results. For women, in general, the younger the woman is the more they can squat; the older the woman is the less they can squat. For men and women, with the 1 exception for women's squat, the peak age of lifting power is between the ages 24-49, at which point lifting power slowly declines. Women's peak performance declines faster than men's peak performance. Women seem to reach their peak sooner than men and decline sooner than men. We also analyzed match attendance. At matches with a large number of competitors, there is a 1:1.7 ratio of women to men, approximately a 2-3 ratio of women to men. Except for the lightest weight category of men, the ratio of weight to lift decreases the more they weigh. For example, a lighter person can generally lift a greater percentage of their weight than a heavier person. In addition, men in general can lift a heavier ratio of their weight when compared with women. The powerlifting stereotype of mostly heavy men lifting extremely large amounts of weights is simply wrong. There is a large amount of variation in age, weight, and sex.

A new algorithmic approach for the extraction of temporal associations from clinical narratives with an application to medical product safety surveillance reports.

The sheer volume of textual information that needs to be reviewed and analyzed in many clinical settings requires the automated retrieval of key clinical and temporal information. The existing natural language processing systems are often challenged by the low quality of clinical texts and do not demonstrate the required performance. In this study, we focus on medical product safety report narratives and investigate the association of the clinical events with appropriate time information. We developed a novel algorithm for tagging and extracting temporal information from the narratives, and associating it with related events. The proposed algorithm minimizes the performance dependency on text quality by relying only on shallow syntactic information and primitive properties of the extracted event and time entities. We demonstrated the effectiveness of the proposed algorithm by evaluating its tagging and time assignment capabilities on 140 randomly selected reports from the US Vaccine Adverse Event Reporting System (VAERS) and the FDA (Food and Drug Administration) Adverse Event Reporting System (FAERS). We compared the performance of our tagger with the SUTime and HeidelTime taggers, and our algorithm's event-time associations with the Temporal Awareness and Reasoning Systems for Question Interpretation (TARSQI). We further evaluated the ability of our algorithm to correctly identify the time information for the events in the 2012 Informatics for Integrating Biology and the Bedside (i2b2) Challenge corpus. For the time tagging task, our algorithm performed better than the SUTime and the HeidelTime taggers (F-measure in VAERS and FAERS: Our algorithm: 0.86 and 0.88, SUTime: 0.77 and 0.74, and HeidelTime 0.75 and 0.42, respectively). In the event-time association task, our algorithm assigned an inappropriate timestamp for 25% of the events, while the TARSQI toolkit demonstrated a considerably lower performance, assigning inappropriate timestamps in 61.5% of the same events. Our algorithm also supported the correct calculation of 69% of the event relations to the section time in the i2b2 testing set.