RESUMEN
Importance: The Sentinel System is a key component of the US Food and Drug Administration (FDA) postmarketing safety surveillance commitment and uses clinical health care data to conduct analyses to inform drug labeling and safety communications, FDA advisory committee meetings, and other regulatory decisions. However, observational data are frequently deemed insufficient for reliable evaluation of safety concerns owing to limitations in underlying data or methodology. Advances in large language models (LLMs) provide new opportunities to address some of these limitations. However, careful consideration is necessary for how and where LLMs can be effectively deployed for these purposes. Observations: LLMs may provide new avenues to support signal-identification activities to identify novel adverse event signals from narrative text of electronic health records. These algorithms may be used to support epidemiologic investigations examining the causal relationship between exposure to a medical product and an adverse event through development of probabilistic phenotyping of health outcomes of interest and extraction of information related to important confounding factors. LLMs may perform like traditional natural language processing tools by annotating text with controlled vocabularies with additional tailored training activities. LLMs offer opportunities for enhancing information extraction from adverse event reports, medical literature, and other biomedical knowledge sources. There are several challenges that must be considered when leveraging LLMs for postmarket surveillance. Prompt engineering is needed to ensure that LLM-extracted associations are accurate and specific. LLMs require extensive infrastructure to use, which many health care systems lack, and this can impact diversity, equity, and inclusion, and result in obscuring significant adverse event patterns in some populations. LLMs are known to generate nonfactual statements, which could lead to false positive signals and downstream evaluation activities by the FDA and other entities, incurring substantial cost. Conclusions and Relevance: LLMs represent a novel paradigm that may facilitate generation of information to support medical product postmarket surveillance activities that have not been possible. However, additional work is required to ensure LLMs can be used in a fair and equitable manner, minimize false positive findings, and support the necessary rigor of signal detection needed for regulatory activities.
Asunto(s)
Procesamiento de Lenguaje Natural , Vigilancia de Productos Comercializados , United States Food and Drug Administration , Vigilancia de Productos Comercializados/métodos , Humanos , Estados Unidos , Registros Electrónicos de SaludRESUMEN
Objectives: The aim of this study was to assess the completeness and readability of generative pre-trained transformer-4 (GPT-4)-generated discharge instructions at prespecified reading levels for common pediatric emergency room complaints. Materials and Methods: The outputs for 6 discharge scenarios stratified by reading level (fifth or eighth grade) and language (English, Spanish) were generated fivefold using GPT-4. Specifically, 120 discharge instructions were produced and analyzed (6 scenarios: 60 in English, 60 in Spanish; 60 at a fifth-grade reading level, 60 at an eighth-grade reading level) and compared for completeness and readability (between language, between reading level, and stratified by group and reading level). Completeness was defined as the proportion of literature-derived key points included in discharge instructions. Readability was quantified using Flesch-Kincaid (English) and Fernandez-Huerta (Spanish) readability scores. Results: English-language GPT-generated discharge instructions contained a significantly higher proportion of must-include discharge instructions than those in Spanish (English: mean (standard error of the mean) = 62% (3%), Spanish: 53% (3%), P = .02). In the fifth-grade and eighth-grade level conditions, there was no significant difference between English and Spanish outputs in completeness. Readability did not differ across languages. Discussion: GPT-4 produced readable discharge instructions in English and Spanish while modulating document reading level. Discharge instructions in English tended to have higher completeness than those in Spanish. Conclusion: Future research in prompt engineering and GPT-4 performance, both generally and in multiple languages, is needed to reduce potential for health disparities by language and reading level.
RESUMEN
OBJECTIVE: Treatment-resistant depression (TRD) occurs in roughly one-third of all individuals with major depressive disorder (MDD). Although research has suggested a significant common variant genetic component of liability to TRD, with heritability estimated at 8% when compared with non-treatment-resistant MDD, no replicated genetic loci have been identified, and the genetic architecture of TRD remains unclear. A key barrier to this work has been the paucity of adequately powered cohorts for investigation, largely because of the challenge in prospectively investigating this phenotype. The objective of this study was to perform a well-powered genetic study of TRD. METHODS: Using receipt of electroconvulsive therapy (ECT) as a surrogate for TRD, the authors applied standard machine learning methods to electronic health record data to derive predicted probabilities of receiving ECT. These probabilities were then applied as a quantitative trait in a genome-wide association study of 154,433 genotyped patients across four large biobanks. RESULTS: Heritability estimates ranged from 2% to 4.2%, and significant genetic overlap was observed with cognition, attention deficit hyperactivity disorder, schizophrenia, alcohol and smoking traits, and body mass index. Two genome-wide significant loci were identified, both previously implicated in metabolic traits, suggesting shared biology and potential pharmacological implications. CONCLUSIONS: This work provides support for the utility of estimation of disease probability for genomic investigation and provides insights into the genetic architecture and biology of TRD.
Asunto(s)
Trastorno Depresivo Mayor , Trastorno Depresivo Resistente al Tratamiento , Terapia Electroconvulsiva , Estudio de Asociación del Genoma Completo , Humanos , Trastorno Depresivo Resistente al Tratamiento/genética , Trastorno Depresivo Resistente al Tratamiento/terapia , Femenino , Masculino , Trastorno Depresivo Mayor/genética , Trastorno Depresivo Mayor/terapia , Persona de Mediana Edad , Aprendizaje Automático , Adulto , Fenotipo , Anciano , Índice de Masa Corporal , Esquizofrenia/genética , Esquizofrenia/terapiaRESUMEN
Suicide prevention requires risk identification, appropriate intervention, and follow-up. Traditional risk identification relies on patient self-reporting, support network reporting, or face-to-face screening with validated instruments or history and physical exam. In the last decade, statistical risk models have been studied and more recently deployed to augment clinical judgment. Models have generally been found to be low precision or problematic at scale due to low incidence. Few have been tested in clinical practice, and none have been tested in clinical trials to our knowledge. Methods: We report the results of a pragmatic randomized controlled trial (RCT) in three outpatient adult Neurology clinic settings. This two-arm trial compared the effectiveness of Interruptive and Non-Interruptive Clinical Decision Support (CDS) to prompt further screening of suicidal ideation for those predicted to be high risk using a real-time, validated statistical risk model of suicide attempt risk, with the decision to screen as the primary end point. Secondary outcomes included rates of suicidal ideation and attempts in both arms. Manual chart review of every trial encounter was used to determine if suicide risk assessment was subsequently documented. Results: From August 16, 2022, through February 16, 2023, our study randomized 596 patient encounters across 561 patients for providers to receive either Interruptive or Non-Interruptive CDS in a 1:1 ratio. Adjusting for provider cluster effects, Interruptive CDS led to significantly higher numbers of decisions to screen (42%=121/289 encounters) compared to Non-Interruptive CDS (4%=12/307) (odds ratio=17.7, p-value <0.001). Secondarily, no documented episodes of suicidal ideation or attempts occurred in either arm. While the proportion of documented assessments among those noting the decision to screen was higher for providers in the Non-Interruptive arm (92%=11/12) than in the Interruptive arm (52%=63/121), the interruptive CDS was associated with more frequent documentation of suicide risk assessment (63/289 encounters compared to 11/307, p-value<0.001). Conclusions: In this pragmatic RCT of real-time predictive CDS to guide suicide risk assessment, Interruptive CDS led to higher numbers of decisions to screen and documented suicide risk assessments. Well-powered large-scale trials randomizing this type of CDS compared to standard of care are indicated to measure effectiveness in reducing suicidal self-harm. ClinicalTrials.gov Identifier: NCT05312437.
RESUMEN
Bipolar disorder is a leading contributor to disability, premature mortality, and suicide. Early identification of risk for bipolar disorder using generalizable predictive models trained on diverse cohorts around the United States could improve targeted assessment of high risk individuals, reduce misdiagnosis, and improve the allocation of limited mental health resources. This observational case-control study intended to develop and validate generalizable predictive models of bipolar disorder as part of the multisite, multinational PsycheMERGE Network across diverse and large biobanks with linked electronic health records (EHRs) from three academic medical centers: in the Northeast (Massachusetts General Brigham), the Mid-Atlantic (Geisinger) and the Mid-South (Vanderbilt University Medical Center). Predictive models were developed and valid with multiple algorithms at each study site: random forests, gradient boosting machines, penalized regression, including stacked ensemble learning algorithms combining them. Predictors were limited to widely available EHR-based features agnostic to a common data model including demographics, diagnostic codes, and medications. The main study outcome was bipolar disorder diagnosis as defined by the International Cohort Collection for Bipolar Disorder, 2015. In total, the study included records for 3,529,569 patients including 12,533 cases (0.3%) of bipolar disorder. After internal and external validation, algorithms demonstrated optimal performance in their respective development sites. The stacked ensemble achieved the best combination of overall discrimination (AUC = 0.82-0.87) and calibration performance with positive predictive values above 5% in the highest risk quantiles at all three study sites. In conclusion, generalizable predictive models of risk for bipolar disorder can be feasibly developed across diverse sites to enable precision medicine. Comparison of a range of machine learning methods indicated that an ensemble approach provides the best performance overall but required local retraining. These models will be disseminated via the PsycheMERGE Network website.
Asunto(s)
Trastorno Bipolar , Humanos , Trastorno Bipolar/diagnóstico , Estudios de Casos y Controles , Medición de Riesgo/métodos , Aprendizaje Automático , Registros Electrónicos de SaludRESUMEN
PURPOSE: Accidental death is a leading cause of mortality among military members and Veterans; however, knowledge is limited regarding time-dependent risk following deployment and if there are differences by type of accidental death. METHODS: Longitudinal cohort study (N = 860,930) of soldiers returning from Afghanistan/Iraq deployments in fiscal years 2008-2014. Accidental deaths (i.e., motor vehicle accidents [MVA], accidental overdose, other accidental deaths), were identified through 2018. Crude and age-adjusted mortality rates, rate ratios, time-dependent hazard rates and trends postdeployment were compared across demographic and military characteristics. RESULTS: During the postdeployment observation period, over one-third of deaths were accidental; most were MVA (46.0 %) or overdoses (37.9 %). Across accidental mortality categories (all, MVA, overdose), younger soldiers (18-24, 25-29) were at higher risk compared to older soldiers (40+), and females at lower risk than males. MVA death rates were highest immediately postdeployment, with a significant decreasing hazard rate over time (annual percent change [APC]: -6.5 %). Conversely, accidental overdose death rates were lowest immediately following deployment, with a significant increasing hazard rate over time (APC: 9.9 %). CONCLUSIONS: Observed divergent trends in risk for the most common types of accidental deaths provide essential information to inform prevention and intervention planning for the immediate postdeployment transition and long-term.
Asunto(s)
Personal Militar , Veteranos , Masculino , Femenino , Humanos , Estudios Longitudinales , Irak , Afganistán , Guerra de Irak 2003-2011RESUMEN
Post marketing safety surveillance depends in part on the ability to detect concerning clinical events at scale. Spontaneous reporting might be an effective component of safety surveillance, but it requires awareness and understanding among healthcare professionals to achieve its potential. Reliance on readily available structured data such as diagnostic codes risk under-coding and imprecision. Clinical textual data might bridge these gaps, and natural language processing (NLP) has been shown to aid in scalable phenotyping across healthcare records in multiple clinical domains. In this study, we developed and validated a novel incident phenotyping approach using unstructured clinical textual data agnostic to Electronic Health Record (EHR) and note type. It's based on a published, validated approach (PheRe) used to ascertain social determinants of health and suicidality across entire healthcare records. To demonstrate generalizability, we validated this approach on two separate phenotypes that share common challenges with respect to accurate ascertainment: 1) suicide attempt; 2) sleep-related behaviors. With samples of 89,428 records and 35,863 records for suicide attempt and sleep-related behaviors, respectively, we conducted silver standard (diagnostic coding) and gold standard (manual chart review) validation. We showed Area Under the Precision-Recall Curve of â¼ 0.77 (95% CI 0.75-0.78) for suicide attempt and AUPR â¼ 0.31 (95% CI 0.28-0.34) for sleep-related behaviors. We also evaluated performance by coded race and demonstrated differences in performance by race were dissimilar across phenotypes and require algorithmovigilance and debiasing prior to implementation.
RESUMEN
Importance: Suicide remains an ongoing concern in the US military. Statistical models have not been broadly disseminated for US Navy service members. Objective: To externally validate and update a statistical suicide risk model initially developed in a civilian setting with an emphasis on primary care. Design, Setting, and Participants: This retrospective cohort study used data collected from 2007 through 2017 among active-duty US Navy service members. The external civilian model was applied to every visit at Naval Medical Center Portsmouth (NMCP), its NMCP Naval Branch Health Clinics (NBHCs), and TRICARE Prime Clinics (TPCs) that fall within the NMCP area. The model was retrained and recalibrated using visits to NBHCs and TPCs and updated using Department of Defense (DoD)-specific billing codes and demographic characteristics, including expanded race and ethnicity categories. Domain and temporal analyses were performed with bootstrap validation. Data analysis was performed from September 2020 to December 2022. Exposure: Visit to US NMCP. Main Outcomes and Measures: Recorded suicidal behavior on the day of or within 30 days of a visit. Performance was assessed using area under the receiver operating curve (AUROC), area under the precision recall curve (AUPRC), Brier score, and Spiegelhalter z-test statistic. Results: Of the 260â¯583 service members, 6529 (2.5%) had a recorded suicidal behavior, 206â¯412 (79.2%) were male; 104â¯835 (40.2%) were aged 20 to 24 years; and 9458 (3.6%) were Asian, 56â¯715 (21.8%) were Black or African American, and 158â¯277 (60.7%) were White. Applying the civilian-trained model resulted in an AUROC of 0.77 (95% CI, 0.74-0.79) and an AUPRC of 0.004 (95% CI, 0.003-0.005) at NBHCs with poor calibration (Spiegelhalter P < .001). Retraining the algorithm improved AUROC to 0.92 (95% CI, 0.91-0.93) and AUPRC to 0.66 (95% CI, 0.63-0.68). Number needed to screen in the top risk tiers was 366 for the external model and 200 for the retrained model; the lower number indicates better performance. Domain validation showed AUROC of 0.90 (95% CI, 0.90-0.91) and AUPRC of 0.01 (95% CI, 0.01-0.01), and temporal validation showed AUROC of 0.75 (95% CI, 0.72-0.78) and AUPRC of 0.003 (95% CI, 0.003-0.005). Conclusions and Relevance: In this cohort study of active-duty Navy service members, a civilian suicide attempt risk model was externally validated. Retraining and updating with DoD-specific variables improved performance. Domain and temporal validation results were similar to external validation, suggesting that implementing an external model in US Navy primary care clinics may bypass the need for costly internal development and expedite the automation of suicide prevention in these clinics.
Asunto(s)
Modelos Estadísticos , Intento de Suicidio , Humanos , Masculino , Femenino , Estudios de Cohortes , Estudios Retrospectivos , Atención Primaria de SaludRESUMEN
The value of genetic information for improving the performance of clinical risk prediction models has yielded variable conclusions. Many methodological decisions have the potential to contribute to differential results across studies. Here, we performed multiple modeling experiments integrating clinical and demographic data from electronic health records (EHR) and genetic data to understand which decision points may affect performance. Clinical data in the form of structured diagnostic codes, medications, procedural codes, and demographics were extracted from two large independent health systems and polygenic risk scores (PRS) were generated across all patients with genetic data in the corresponding biobanks. Crohn's disease was used as the model phenotype based on its substantial genetic component, established EHR-based definition, and sufficient prevalence for model training and testing. We investigated the impact of PRS integration method, as well as choices regarding training sample, model complexity, and performance metrics. Overall, our results show that including PRS resulted in higher performance by some metrics but the gain in performance was only robust when combined with demographic data alone. Improvements were inconsistent or negligible after including additional clinical information. The impact of genetic information on performance also varied by PRS integration method, with a small improvement in some cases from combining PRS with the output of a clinical model (late-fusion) compared to its inclusion an additional feature (early-fusion). The effects of other modeling decisions varied between institutions though performance increased with more compute-intensive models such as random forest. This work highlights the importance of considering methodological decision points in interpreting the impact on prediction performance when including PRS information in clinical models.
RESUMEN
Purpose: To estimate the association of psychiatric polygenic scores with healthcare utilization and comorbidity burden. Methods: Observational cohort study (N = 118,882) of adolescent and adult biobank participants with linked electronic health records (EHRs) from three diverse study sites; (Massachusetts General Brigham, Vanderbilt University Medical Center, Geisinger). Polygenic scores (PGS) were derived from the largest available GWAS of major depressive depression, bipolar disorder, and schizophrenia at the time of analysis. Negative binomial regression models were used to estimate the association between each psychiatric PGS and healthcare utilization and comorbidity burden. Healthcare utilization was measured as frequency of emergency department (ED), inpatient (IP), and outpatient (OP) visits. Comorbidity burden was defined by the Elixhauser Comorbidity Index and the Charlson Comorbidity Index. Results: Participants had a median follow-up duration of 12 years in the EHR. Individuals in the top decile of polygenic score for major depressive disorder had significantly more ED visits (RR=1.22, 95% CI; 1.17, 1.29) compared to those the lowest decile. Increases were also observed with IP and comorbidity burden. Among those diagnosed with depression and in the highest decile of the PGS, there was an increase in all utilization types (ED: RR=1.56, 95% CI 1.41, 1.72; OP: RR=1.16, 95% CI 1.08, 1.24; IP: RR=1.23, 95% CI 1.12, 1.36) post-diagnosis. No clinically significant results were observed with bipolar and schizophrenia polygenic scores. Conclusions: Polygenic score for depression is modestly associated with increased healthcare resource utilization and comorbidity burden, in the absence of diagnosis. Following a diagnosis of depression, the PGS was associated with further increases in healthcare utilization. These findings suggest that depression genetic risk is associated with utilization and burden of chronic disease in real-world settings.
RESUMEN
Objectives: We evaluated autoencoders as a feature engineering and pretraining technique to improve major depressive disorder (MDD) prognostic risk prediction. Autoencoders can represent temporal feature relationships not identified by aggregate features. The predictive performance of autoencoders of multiple sequential structures was evaluated as feature engineering and pretraining strategies on an array of prediction tasks and compared to a restricted Boltzmann machine (RBM) and random forests as a benchmark. Materials and Methods: We study MDD patients from Vanderbilt University Medical Center. Autoencoder models with Attention and long-short-term memory (LSTM) layers were trained to create latent representations of the input data. Predictive performance was evaluated temporally by fitting random forest models to predict future outcomes with engineered features as input and using autoencoder weights to initialize neural network layers. We evaluated area under the precision-recall curve (AUPRC) trends and variation over the study population's treatment course. Results: The pretrained LSTM model improved predictive performance over pretrained Attention models and benchmarks in 3 of 4 outcomes including self-harm/suicide attempt (AUPRCs, LSTM pretrained = 0.012, Attention pretrained = 0.010, RBM = 0.009, random forest = 0.005). The use of autoencoders for feature engineering had varied results, with benchmarks outperforming LSTM and Attention encodings on the self-harm/suicide attempt outcome (AUPRCs, LSTM encodings = 0.003, Attention encodings = 0.004, RBM = 0.009, random forest = 0.005). Discussion: Improvement in prediction resulting from pretraining has the potential for increased clinical impact of MDD risk models. We did not find evidence that the use of temporal feature encodings was additive to predictive performance in the study population. This suggests that predictive information retained by model weights may be lost during encoding. LSTM pretrained model predictive performance is shown to be clinically useful and improves over state-of-the-art predictors in the MDD phenotype. LSTM model performance warrants consideration of use in future related studies. Conclusion: LSTM models with pretrained weights from autoencoders were able to outperform the benchmark and a pretrained Attention model. Future researchers developing risk models in MDD may benefit from the use of LSTM autoencoder pretrained weights.
RESUMEN
Importance: Research to identify the direct and indirect associations of military-related traumatic brain injury (TBI) with suicide has been complicated by a range of data-related challenges. Objective: To identify differences in rates of new-onset mental health conditions (ie, anxiety, mood, posttraumatic stress, adjustment, alcohol use, and substance use disorders) among soldiers with and without a history of military-related TBI and to explore the direct and indirect (through new-onset mental health disorders) associations of TBI with suicide. Design, Setting, and Participants: This retrospective cohort study used data from the Substance Use and Psychological Injury Combat Study (SUPIC) database. Demographic, military, and health data from the Department of Defense within SUPIC were compiled and linked with National Death Index records to identify deaths by suicide. Participants included US Army soldiers who returned from an Afghanistan or Iraq deployment. Data were analyzed from September to December 2022. Exposures: Military-related TBI. Main Outcomes and Measures: The outcome of interest was suicide. Secondary outcomes were incidence of new-onset mental health conditions. Mediation analyses consisted of accelerated failure time (AFT) models in conjunction with the product of coefficients method. The 6 new-onset mental health diagnosis categories and the 2 or more categories variable were each considered separately as potential mediators; therefore, a total of 14 models plus the overall AFT model estimating the total effect associated with TBI in suicide risk were fit. Results: The study included 860â¯892 soldiers (320â¯539 soldiers [37.2%] aged 18-24 at end of index deployment; 766â¯454 [89.0%] male), with 108â¯785 soldiers (12.6%) with at least 1 documented TBI on their military health record. Larger increases in mental health diagnoses were observed for all conditions from before to after documented TBI, compared with the matched dates for those without a history of TBI, with increases observed for mood (67.7% vs 37.5%) and substance use (100% vs 14.5%). Time-to-suicide direct effect estimates for soldiers with a history of TBI were similar across mediators. For example, considering new-onset adjustment disorders, time-to-suicide was 16.7% faster (deceleration factor, 0.833; 95% CI, 0.756-0.912) than for soldiers without a history of TBI. Indirect effect estimates of associations with TBI were substantial and varied across mediators. The largest indirect effect estimate was observed through the association with new-onset substance use disorder, with a time to suicide 63.8% faster (deceleration factor, 0.372; 95% CI, 0.322-0.433) for soldiers with a history of TBI. Conclusions and Relevance: In this longitudinal cohort study of soldiers, rates of new-onset mental health conditions were higher among individuals with a history of TBI compared with those without. Moreover, risk for suicide was both directly and indirectly associated with history of TBI. These findings suggest that increased efforts are needed to conceptualize the accumulation of risk associated with multiple military-related exposures and identify evidence-based interventions that address mechanisms associated with frequently co-occurring conditions.
Asunto(s)
Lesiones Traumáticas del Encéfalo , Personal Militar , Suicidio , Femenino , Humanos , Masculino , Lesiones Traumáticas del Encéfalo/epidemiología , Estudios Longitudinales , Salud Mental , Estudios Retrospectivos , Estados Unidos/epidemiología , Adolescente , Adulto JovenRESUMEN
Importance: There are many prognostic models of suicide risk, but few have been prospectively evaluated, and none has been developed specifically for Native American populations. Objective: To prospectively validate a statistical risk model implemented in a community setting and evaluate whether use of this model was associated with improved reach of evidence-based care and reduced subsequent suicide-related behavior among high-risk individuals. Design, Setting, and Participants: This prognostic study, done in partnership with the White Mountain Apache Tribe, used data collected by the Apache Celebrating Life program for adults aged 25 years or older identified as at risk for suicide and/or self-harm from January 1, 2017, through August 31, 2022. Data were divided into 2 cohorts: (1) individuals and suicide-related events from the period prior to suicide risk alerts being active (February 29, 2020) and (2) individuals and events from the time after alerts were activated. Main Outcomes and Measures: Aim 1 focused on a prospective validation of the risk model in cohort 1. Aim 2 compared the odds of repeated suicide-related events and the reach of brief contact interventions among high-risk cases between cohort 2 and cohort 1. Results: Across both cohorts, a total of 400 individuals identified as at risk for suicide and/or self-harm (mean [SD] age, 36.5 [10.3] years; 210 females [52.5%]) had 781 suicide-related events. Cohort 1 included 256 individuals with index events prior to active notifications. Most index events (134 [52.5%]) were for binge substance use, followed by 101 (39.6%) for suicidal ideation, 28 (11.0%) for a suicide attempt, and 10 (3.9%) for self-injury. Among these individuals, 102 (39.5%) had subsequent suicidal behaviors. In cohort 1, the majority (220 [86.3%]) were classified as low risk, and 35 individuals (13.3%) were classified as high risk for suicidal attempt or death in the 12 months after their index event. Cohort 2 included 144 individuals with index events after notifications were activated. For aim 1, those classified as high risk had a greater odds of subsequent suicide-related events compared with those classified as low risk (odds ratio [OR], 3.47; 95% CI, 1.53-7.86; P = .003; area under the receiver operating characteristic curve, 0.65). For aim 2, which included 57 individuals classified as high risk across both cohorts, during the time when alerts were inactive, high-risk individuals were more likely to have subsequent suicidal behaviors compared with when alerts were active (OR, 9.14; 95% CI, 1.85-45.29; P = .007). Before the active alerts, only 1 of 35 (2.9%) individuals classified as high risk received a wellness check; after the alerts were activated, 11 of 22 (50.0%) individuals classified as high risk received 1 or more wellness checks. Conclusions and Relevance: This study showed that a statistical model and associated care system developed in partnership with the White Mountain Apache Tribe enhanced identification of individuals at high risk for suicide and was associated with a reduced risk for subsequent suicidal behaviors and increased reach of care.
Asunto(s)
Indio Americano o Nativo de Alaska , Conducta Autodestructiva , Adulto , Femenino , Humanos , Conducta Autodestructiva/diagnóstico , Conducta Autodestructiva/epidemiología , Conducta Autodestructiva/etnología , Conducta Autodestructiva/prevención & control , Ideación Suicida , Intento de Suicidio/etnología , Intento de Suicidio/prevención & control , Intento de Suicidio/estadística & datos numéricos , Medición de Riesgo/etnología , Medición de Riesgo/estadística & datos numéricos , Suicidio/etnología , Suicidio/psicología , Suicidio/estadística & datos numéricos , Pronóstico , Modelos EstadísticosRESUMEN
BACKGROUND: Fatal drug overdose surveillance informs prevention but is often delayed because of autopsy report processing and death certificate coding. Autopsy reports contain narrative text describing scene evidence and medical history (similar to preliminary death scene investigation reports) and may serve as early data sources for identifying fatal drug overdoses. To facilitate timely fatal overdose reporting, natural language processing was applied to narrative texts from autopsies. OBJECTIVE: This study aimed to develop a natural language processing-based model that predicts the likelihood that an autopsy report narrative describes an accidental or undetermined fatal drug overdose. METHODS: Autopsy reports of all manners of death (2019-2021) were obtained from the Tennessee Office of the State Chief Medical Examiner. The text was extracted from autopsy reports (PDFs) using optical character recognition. Three common narrative text sections were identified, concatenated, and preprocessed (bag-of-words) using term frequency-inverse document frequency scoring. Logistic regression, support vector machine (SVM), random forest, and gradient boosted tree classifiers were developed and validated. Models were trained and calibrated using autopsies from 2019 to 2020 and tested using those from 2021. Model discrimination was evaluated using the area under the receiver operating characteristic, precision, recall, F1-score, and F2-score (prioritizes recall over precision). Calibration was performed using logistic regression (Platt scaling) and evaluated using the Spiegelhalter z test. Shapley additive explanations values were generated for models compatible with this method. In a post hoc subgroup analysis of the random forest classifier, model discrimination was evaluated by forensic center, race, age, sex, and education level. RESULTS: A total of 17,342 autopsies (n=5934, 34.22% cases) were used for model development and validation. The training set included 10,215 autopsies (n=3342, 32.72% cases), the calibration set included 538 autopsies (n=183, 34.01% cases), and the test set included 6589 autopsies (n=2409, 36.56% cases). The vocabulary set contained 4002 terms. All models showed excellent performance (area under the receiver operating characteristic ≥0.95, precision ≥0.94, recall ≥0.92, F1-score ≥0.94, and F2-score ≥0.92). The SVM and random forest classifiers achieved the highest F2-scores (0.948 and 0.947, respectively). The logistic regression and random forest were calibrated (P=.95 and P=.85, respectively), whereas the SVM and gradient boosted tree classifiers were miscalibrated (P=.03 and P<.001, respectively). "Fentanyl" and "accident" had the highest Shapley additive explanations values. Post hoc subgroup analyses revealed lower F2-scores for autopsies from forensic centers D and E. Lower F2-score were observed for the American Indian, Asian, ≤14 years, and ≥65 years subgroups, but larger sample sizes are needed to validate these findings. CONCLUSIONS: The random forest classifier may be suitable for identifying potential accidental and undetermined fatal overdose autopsies. Further validation studies should be conducted to ensure early detection of accidental and undetermined fatal drug overdoses across all subgroups.
Asunto(s)
Sobredosis de Droga , Procesamiento de Lenguaje Natural , Humanos , Autopsia , Algoritmos , Bosques AleatoriosRESUMEN
Bipolar disorder is a leading contributor to disability, premature mortality, and suicide. Early identification of risk for bipolar disorder using generalizable predictive models trained on diverse cohorts around the United States could improve targeted assessment of high risk individuals, reduce misdiagnosis, and improve the allocation of limited mental health resources. This observational case-control study intended to develop and validate generalizable predictive models of bipolar disorder as part of the multisite, multinational PsycheMERGE Consortium across diverse and large biobanks with linked electronic health records (EHRs) from three academic medical centers: in the Northeast (Massachusetts General Brigham), the Mid-Atlantic (Geisinger) and the Mid-South (Vanderbilt University Medical Center). Predictive models were developed and validated with multiple algorithms at each study site: random forests, gradient boosting machines, penalized regression, including stacked ensemble learning algorithms combining them. Predictors were limited to widely available EHR-based features agnostic to a common data model including demographics, diagnostic codes, and medications. The main study outcome was bipolar disorder diagnosis as defined by the International Cohort Collection for Bipolar Disorder, 2015. In total, the study included records for 3,529,569 patients including 12,533 cases (0.3%) of bipolar disorder. After internal and external validation, algorithms demonstrated optimal performance in their respective development sites. The stacked ensemble achieved the best combination of overall discrimination (AUC = 0.82 - 0.87) and calibration performance with positive predictive values above 5% in the highest risk quantiles at all three study sites. In conclusion, generalizable predictive models of risk for bipolar disorder can be feasibly developed across diverse sites to enable precision medicine. Comparison of a range of machine learning methods indicated that an ensemble approach provides the best performance overall but required local retraining. These models will be disseminated via the PsycheMERGE Consortium website.
RESUMEN
Efforts were focused on identifying differences in suicide rates and time-dependent hazard rate trends, overall and within age groups, by race and ethnicity among United States Army members who returned from an index deployment (October 2007 to September 2014). This retrospective cohort study was conducted using an existing longitudinal database, the Substance Use and Psychological Injury Combat Study (SUPIC). Demographic (e.g., race and ethnicity) and military data from the Department of Defense compiled within SUPIC, as well as Department of Veterans Affairs data were linked with National Death Index records (through 2018) to identify deaths by suicide including those that occurred after military service. The cohort included 860,930 Army Service members (Active Duty, National Guard, and Reserve). Age-adjusted (using the direct standardization method) and age-specific suicide rates per 100,000 person years were calculated and rate ratios (RR) were used for comparisons. Trends were evaluated using hazard rates over time since the end of individuals' index deployments. Among those aged 18-29 at the end of their index deployment, the suicide rate for American Indian/Alaskan Native (AI/AN) individuals was 1.51 times higher (95% confidence interval [CI]: 1.03, 2.14) compared to White non-Hispanic individuals (WNH), and lower for Hispanic and Black non-Hispanic (BNH) than for WNH individuals (RR = 0.65 [95% CI: 0.55, 0.77] and RR = 0.71 [95% CI: 0.61, 0.82], respectively). However, analyses revealed increasing trends in hazard rates post-deployment (≤ 6.5 years) within groups of Hispanic and BNH individuals (Average Annual Percent Change [APC]: 12.1% [95% CI: 1.3%, 24.1%] and 11.4% [95% CI: 6.9%, 16.0%], respectively) with a smaller, increase for WNH individuals (APC: 3.1%; 95% CI: 0.1%, 6.1%). Findings highlight key subgroups at risk for post-deployment suicide (i.e., WNH, AI/AN and younger individuals), as well as heterogeneous trends overtime, with rates and trends varying within race and ethnic groups by age groups. Post-deployment suicide prevention efforts that address culturally relevant factors and social determinants of health associated with health inequities are needed.
Asunto(s)
Personal Militar , Suicidio , Humanos , Estados Unidos/epidemiología , Etnicidad , Estudios Retrospectivos , Personal Militar/psicología , Prevención del SuicidioRESUMEN
Patients with autism spectrum disorder (ASD) access healthcare frequently, yet little is known about their interactions with patient portals. To describe adults with ASD using patient portal, we conducted regression analyses of visit history, demographics, co-occurring conditions and diagnoses, and patient portal use to determine factors most indicative of whether a patient 1) has sent at least one message (via patient or proxy) and 2) has at least one message sent on their behalf via a proxy account after they turned 18 years old. The 2,412-person cohort had 996 (41.3%) patients who had sent at least one message on their account with 129 (5.3%) of patients having at least one proxy message. This study found that adults with ASD are less likely to use messaging functionality and more likely to have a message sent via proxy than other patient populations. Comorbid mental illness was correlated with using messaging functionality.
Asunto(s)
Trastorno del Espectro Autista , Portales del Paciente , Adulto , Humanos , Adolescente , Pacientes , Atención a la SaludRESUMEN
Cross-validation remains a popular means of developing and validating artificial intelligence for health care. Numerous subtypes of cross-validation exist. Although tutorials on this validation strategy have been published and some with applied examples, we present here a practical tutorial comparing multiple forms of cross-validation using a widely accessible, real-world electronic health care data set: Medical Information Mart for Intensive Care-III (MIMIC-III). This tutorial explored methods such as K-fold cross-validation and nested cross-validation, highlighting their advantages and disadvantages across 2 common predictive modeling use cases: classification (mortality) and regression (length of stay). We aimed to provide readers with reproducible notebooks and best practices for modeling with electronic health care data. We also described sets of useful recommendations as we demonstrated that nested cross-validation reduces optimistic bias but comes with additional computational challenges. This tutorial might improve the community's understanding of these important methods while catalyzing the modeling community to apply these guides directly in their work using the published code.
RESUMEN
Introduction: Opioid use disorders (OUDs) constitute a major public health issue, and we urgently need alternative methods for characterizing risk for OUD. Electronic health records (EHRs) are useful tools for understanding complex medical phenotypes but have been underutilized for OUD because of challenges related to underdiagnosis, binary diagnostic frameworks, and minimally characterized reference groups. As a first step in addressing these challenges, a new paradigm is warranted that characterizes risk for opioid prescription misuse on a continuous scale of severity, i.e., as a continuum. Methods: Across sites within the PsycheMERGE network, we extracted prescription opioid data and diagnoses that co-occur with OUD (including psychiatric and substance use disorders, pain-related diagnoses, HIV, and hepatitis C) for over 2.6 million patients across three health registries (Vanderbilt University Medical Center, Mass General Brigham, Geisinger) between 2005 and 2018. We defined three groups based on levels of opioid exposure: no prescriptions, minimal exposure, and chronic exposure and then compared the comorbidity profiles of these groups to the full registries and to those with OUD diagnostic codes. Results: Our results confirm that EHR data reflects known higher prevalence of substance use disorders, psychiatric disorders, medical, and pain diagnoses in patients with OUD diagnoses and chronic opioid use. Comorbidity profiles that distinguish opioid exposure are strikingly consistent across large health systems, indicating the phenotypes described in this new quantitative framework are robust to health systems differences. Conclusion: This work indicates that EHR prescription opioid data can serve as a platform to characterize complex risk markers for OUD using existing data.