Clozapine Use Among People With Psychotic Disorders Who Experience Specific Indications for Clozapine.

Objective: To examine rates of clozapine use among people with psychotic disorders who experience specific indications for clozapine.Methods: Records data from 11 integrated health systems identified patients aged 18 years or older with recorded International Classification of Diseases, Tenth Revision, Clinical Modification, diagnoses of schizophrenia, schizoaffective disorder, or other psychotic disorder who experienced any of the 3 events between January 1, 2019, and December 31, 2019, suggesting indications for clozapine: a diagnosis of self-harm injury or poisoning, suicidal ideation diagnosed or in response to standardized assessments, and hospitalization or emergency department (ED) care for psychotic disorder despite treatment with 2 or more antipsychotic medications. Prescription dispensing data identified all clozapine use prior to or in the 12 months following each indication event. Analyses were conducted with aggregate data from each health system; no individual data were shared.Results: A total of 7,648 patients with psychotic disorder diagnoses experienced at least 1 indication event. Among 1,097 experiencing a self-harm event, 32 (2.9%) had any prior clozapine use, and 10 (0.9%) initiated clozapine during the following 12 months. Among 6,396 with significant suicidal ideation, 238 (3.7%) had any prior clozapine use, and 70 (1.1%) initiated clozapine over 12 months. Among 881 with hospitalization or ED visit despite pharmacotherapy, 77 (8.7%) had any prior clozapine treatment, and 41 (4.7%) initiated clozapine over 12 months. Among those with significant suicidal ideation, rates of both prior clozapine treatment and subsequent initiation varied significantly by race and ethnicity, with rates among Hispanic and non-Hispanic Black patients lower than among non Hispanic White patients.Conclusions: Initiating clozapine treatment is uncommon among people with psychotic disorders who experience events suggesting clozapine is indicated, with even lower rates among Black and Hispanic patients.

Current use of estrogen-containing oral contraceptives or hormone therapy and risk of COVID-19 infection and hospitalization: a population-based cohort study.

This population-based cohort study evaluated the association between current use of oral contraceptives (OC) among women under 50 years (n=306,541), and hormone therapy (HT) among women aged 50 or older (n=323,203), and COVID-19 infection and hospitalization. Current OC/HT use was recorded monthly using prescription dispensing data. COVID-19 infections were identified March 2020-February 2021. COVID-19 infection and hospitalization were identified through diagnosis codes and laboratory tests. Weighted generalized estimating equations models estimated multivariable-adjusted odds ratios (aORs) for COVID-19 infection associated with time-varying OC/HT use. Among women with COVID-19, logistic regression models evaluated OC/HT use and COVID-19 hospitalization. Over 12 months, 11,727 (3.8%) women <50 years and 8,661 (2.7%) women ≥50 years experienced COVID-19 infections. There was no evidence of an association between OC use and infection (aOR=1.05; 95%CI: 0.97, 1.12). There was a modest association between HT use and infection (aOR=1.19; 95%CI: 1.03, 1.38). Women using OC had a 39% lower risk of hospitalization (aOR=0.61; 95%CI: 0.38, 1.00), but there was no association of HT use with hospitalization (aOR=0.89; 95%CI: 0.51, 1.53). These findings do not suggest a meaningfully greater risk of COVID-19 infection associated with OC or HT use. OC use may be associated with lower COVID-19 hospitalization risk.

Variable selection for individualised treatment rules with discrete outcomes.

An individualised treatment rule (ITR) is a decision rule that aims to improve individuals' health outcomes by recommending treatments according to subject-specific information. In observational studies, collected data may contain many variables that are irrelevant to treatment decisions. Including all variables in an ITR could yield low efficiency and a complicated treatment rule that is difficult to implement. Thus, selecting variables to improve the treatment rule is crucial. We propose a doubly robust variable selection method for ITRs, and show that it compares favourably with competing approaches. We illustrate the proposed method on data from an adaptive, web-based stress management tool.

Predicting suicide death after emergency department visits with mental health or self-harm diagnoses.

OBJECTIVE: Use health records data to predict suicide death following emergency department visits. METHODS: Electronic health records and insurance claims from seven health systems were used to: identify emergency department visits with mental health or self-harm diagnoses by members aged 11 or older; extract approximately 2500 potential predictors including demographic, historical, and baseline clinical characteristics; and ascertain subsequent deaths by self-harm. Logistic regression with lasso and random forest models predicted self-harm death over 90 days after each visit. RESULTS: Records identified 2,069,170 eligible visits, 899 followed by suicide death within 90 days. The best-fitting logistic regression with lasso model yielded an area under the receiver operating curve of 0.823 (95% CI 0.810-0.836). Visits above the 95th percentile of predicted risk included 34.8% (95% CI 31.1-38.7) of subsequent suicide deaths and had a 0.303% (95% CI 0.261-0.346) suicide death rate over the following 90 days. Model performance was similar across subgroups defined by age, sex, race, and ethnicity. CONCLUSIONS: Machine learning models using coded data from health records have moderate performance in predicting suicide death following emergency department visits for mental health or self-harm diagnosis and could be used to identify patients needing more systematic follow-up.

Preventing severe hypoglycemia in adults with type 2 diabetes (PHT2): Design, delivery and evaluation framework for a randomized controlled trial.

BACKGROUND: Severe hypoglycemia is a common and feared complication of medications used to lower blood glucose levels in individuals with diabetes. Psychoeducational interventions can prevent severe hypoglycemia in individuals with type 1 diabetes (T1D). We aim to determine the effectiveness of this approach among adults with type 2 diabetes (T2D) at elevated risk for severe hypoglycemia. METHODS: Preventing Hypoglycemia in Type 2 diabetes (PHT2) is a two-arm, parallel, randomized controlled trial. Participants are eligible if they are adults with T2D receiving care at an integrated group practice in Washington state and have experienced one or more episodes of severe hypoglycemia in the prior 12 months or have impaired awareness of hypoglycemia (Gold score ≥ 4). Participants are randomized to proactive nurse care management with or without my hypo compass, an evidence-based, psychoeducational intervention combining group and individual self-management training. For this study, my hypo compass was adapted to be suitable for adults with T2D and from an in-person to a virtual intervention over videoconference and telephone. The primary outcome is any self-reported severe hypoglycemia in the 12 months following the start of the intervention. Secondary outcomes include biochemical measures of hypoglycemia, self-reported hypoglycemia awareness, fear of hypoglycemia, and emergency department visits and hospitalizations for severe hypoglycemia. The study includes a process evaluation to assess implementation fidelity and clarify the causal pathway. CONCLUSION: The PHT2 trial will compare the effectiveness of two approaches for reducing severe hypoglycemia in adults with T2D. TRIAL REGISTRATION: clinicaltrials.gov, # NCT04863872.

Stability of Suicide Risk Prediction Models During Changes in Health Care Delivery.

OBJECTIVE: The authors aimed to use health records data to examine how the accuracy of statistical models predicting self-harm or suicide changed between 2015 and 2019, as health systems implemented suicide prevention programs. METHODS: Data from four large health systems were used to identify specialty mental health visits by patients ages ≥11 years, assess 311 potential predictors of self-harm (including demographic characteristics, historical risk factors, and index visit characteristics), and ascertain fatal or nonfatal self-harm events over 90 days after each visit. New prediction models were developed with logistic regression with LASSO (least absolute shrinkage and selection operator) in random samples of visits (65%) from each calendar year and were validated in the remaining portion of the sample (35%). RESULTS: A model developed for visits from 2009 to mid-2015 showed similar classification performance and calibration accuracy in a new sample of about 13.1 million visits from late 2015 to 2019. Area under the receiver operating characteristic curve (AUC) ranged from 0.840 to 0.849 in the new sample, compared with 0.851 in the original sample. New models developed for each year for 2015-2019 had classification performance (AUC range 0.790-0.853), sensitivity, and positive predictive value similar to those of the previously developed model. Models selected similar predictors from 2015 to 2019, except for more frequent selection of depression questionnaire data in later years, when questionnaires were more frequently recorded. CONCLUSIONS: A self-harm prediction model developed with 2009-2015 visit data performed similarly when applied to 2015-2019 visits. New models did not yield superior performance or identify different predictors.

Predicting risk of suicidal behavior from insurance claims data vs. linked data from insurance claims and electronic health records.

PURPOSE: Observational studies assessing effects of medical products on suicidal behavior often rely on health record data to account for pre-existing risk. We assess whether high-dimensional models predicting suicide risk using data derived from insurance claims and electronic health records (EHRs) are superior to models using data from insurance claims alone. METHODS: Data were from seven large health systems identified outpatient mental health visits by patients aged 11 or older between 1/1/2009 and 9/30/2017. Data for the 5 years prior to each visit identified potential predictors of suicidal behavior typically available from insurance claims (e.g., mental health diagnoses, procedure codes, medication dispensings) and additional potential predictors available from EHRs (self-reported race and ethnicity, responses to Patient Health Questionnaire or PHQ-9 depression questionnaires). Nonfatal self-harm events following each visit were identified from insurance claims data and fatal self-harm events were identified by linkage to state mortality records. Random forest models predicting nonfatal or fatal self-harm over 90 days following each visit were developed in a 70% random sample of visits and validated in a held-out sample of 30%. Performance of models using linked claims and EHR data was compared to models using claims data only. RESULTS: Among 15 845 047 encounters by 1 574 612 patients, 99 098 (0.6%) were followed by a self-harm event within 90 days. Overall classification performance did not differ between the best-fitting model using all data (area under the receiver operating curve or AUC = 0.846, 95% CI 0.839-0.854) and the best-fitting model limited to data available from insurance claims (AUC = 0.846, 95% CI 0.838-0.853). Competing models showed similar classification performance across a range of cut-points and similar calibration performance across a range of risk strata. Results were similar when the sample was limited to health systems and time periods where PHQ-9 depression questionnaires were recorded more frequently. CONCLUSION: Investigators using health record data to account for pre-existing risk in observational studies of suicidal behavior need not limit that research to databases including linked EHR data.

Predicting Outcomes of Antidepressant Treatment in Community Practice Settings.

OBJECTIVE: The authors examined whether machine-learning models could be used to analyze data from electronic health records (EHRs) to predict patients' responses to antidepressant medications. METHODS: EHR data from a Washington State health system identified patients ages ≥13 years who started an antidepressant medication in 2016 in a community practice setting and had a baseline Patient Health Questionnaire-9 (PHQ-9) score of ≥10 and at least one PHQ-9 score recorded 14-180 days later. Potential predictors of a response to antidepressants were extracted from the EHR and included demographic characteristics, psychiatric and substance use diagnoses, past psychiatric medication use, mental health service use, and past PHQ-9 scores. Random-forest and penalized regression analyses were used to build models predicting follow-up PHQ-9 score and a favorable treatment response (≥50% improvement in score). RESULTS: Among 2,469 patients starting antidepressant medication treatment, the mean±SD baseline PHQ-9 score was 17.3±4.5, and the mean lowest follow-up score was 9.2±5.9. Outcome data were available for 72% of the patients. About 48% of the patients had a favorable treatment response. The best-fitting random-forest models yielded a correlation between predicted and observed follow-up scores of 0.38 (95% CI=0.32-0.45) and an area under the receiver operating characteristic curve for a favorable response of 0.57 (95% CI=0.52-0.61). Results were similar for penalized regression models and for models predicting last PHQ-9 score during follow-up. CONCLUSIONS: Prediction models using EHR data were not accurate enough to inform recommendations for or against starting antidepressant medication. Personalization of depression treatment should instead rely on systematic assessment of early outcomes.

Variable selection in high dimensions for discrete-outcome individualized treatment rules: Reducing severity of depression symptoms.

Despite growing interest in estimating individualized treatment rules, little attention has been given the binary outcome setting. Estimation is challenging with nonlinear link functions, especially when variable selection is needed. We use a new computational approach to solve a recently proposed doubly robust regularized estimating equation to accomplish this difficult task in a case study of depression treatment. We demonstrate an application of this new approach in combination with a weighted and penalized estimating equation to this challenging binary outcome setting. We demonstrate the double robustness of the method and its effectiveness for variable selection. The work is motivated by and applied to an analysis of treatment for unipolar depression using a population of patients treated at Kaiser Permanente Washington.

Monte Carlo sensitivity analysis for unmeasured confounding in dynamic treatment regimes.

Data-driven methods for personalizing treatment assignment have garnered much attention from clinicians and researchers. Dynamic treatment regimes formalize this through a sequence of decision rules that map individual patient characteristics to a recommended treatment. Observational studies are commonly used for estimating dynamic treatment regimes due to the potentially prohibitive costs of conducting sequential multiple assignment randomized trials. However, estimating a dynamic treatment regime from observational data can lead to bias in the estimated regime due to unmeasured confounding. Sensitivity analyses are useful for assessing how robust the conclusions of the study are to a potential unmeasured confounder. A Monte Carlo sensitivity analysis is a probabilistic approach that involves positing and sampling from distributions for the parameters governing the bias. We propose a method for performing a Monte Carlo sensitivity analysis of the bias due to unmeasured confounding in the estimation of dynamic treatment regimes. We demonstrate the performance of the proposed procedure with a simulation study and apply it to an observational study examining tailoring the use of antidepressant medication for reducing symptoms of depression using data from Kaiser Permanente Washington.

Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction.

Suicide risk prediction models can identify individuals for targeted intervention. Discussions of transparency, explainability, and transportability in machine learning presume complex prediction models with many variables outperform simpler models. We compared random forest, artificial neural network, and ensemble models with 1500 temporally defined predictors to logistic regression models. Data from 25,800,888 mental health visits made by 3,081,420 individuals in 7 health systems were used to train and evaluate suicidal behavior prediction models. Model performance was compared across several measures. All models performed well (area under the receiver operating curve [AUC]: 0.794-0.858). Ensemble models performed best, but improvements over a regression model with 100 predictors were minimal (AUC improvements: 0.006-0.020). Results are consistent across performance metrics and subgroups defined by race, ethnicity, and sex. Our results suggest simpler parametric models, which are easier to implement as part of routine clinical practice, perform comparably to more complex machine learning methods.

Estimating individualized treatment rules in longitudinal studies with covariate-driven observation times.

The sequential treatment decisions made by physicians to treat chronic diseases are formalized in the statistical literature as dynamic treatment regimes. To date, methods for dynamic treatment regimes have been developed under the assumption that observation times, that is, treatment and outcome monitoring times, are determined by study investigators. That assumption is often not satisfied in electronic health records data in which the outcome, the observation times, and the treatment mechanism are associated with patients' characteristics. The treatment and observation processes can lead to spurious associations between the treatment of interest and the outcome to be optimized under the dynamic treatment regime if not adequately considered in the analysis. We address these associations by incorporating two inverse weights that are functions of a patient's covariates into dynamic weighted ordinary least squares to develop optimal single stage dynamic treatment regimes, known as individualized treatment rules. We show empirically that our methodology yields consistent, multiply robust estimators. In a cohort of new users of antidepressant drugs from the United Kingdom's Clinical Practice Research Datalink, the proposed method is used to develop an optimal treatment rule that chooses between two antidepressants to optimize a utility function related to the change in body mass index.

Association Between Diabetes Severity and Risks of COVID-19 Infection and Outcomes.

BACKGROUND: Little is known about whether diabetes increases the risk of COVID-19 infection and whether measures of diabetes severity are related to COVID-19 outcomes. OBJECTIVE: Investigate diabetes severity measures as potential risk factors for COVID-19 infection and COVID-19 outcomes. DESIGN, PARTICIPANTS, MEASURES: In integrated healthcare systems in Colorado, Oregon, and Washington, we identified a cohort of adults on February 29, 2020 (n = 1,086,918) and conducted follow-up through February 28, 2021. Electronic health data and death certificates were used to identify markers of diabetes severity, covariates, and outcomes. Outcomes were COVID-19 infection (positive nucleic acid antigen test, COVID-19 hospitalization, or COVID-19 death) and severe COVID-19 (invasive mechanical ventilation or COVID-19 death). Individuals with diabetes (n = 142,340) and categories of diabetes severity measures were compared with a referent group with no diabetes (n = 944,578), adjusting for demographic variables, neighborhood deprivation index, body mass index, and comorbidities. RESULTS: Of 30,935 patients with COVID-19 infection, 996 met the criteria for severe COVID-19. Type 1 (odds ratio [OR] 1.41, 95% CI 1.27-1.57) and type 2 diabetes (OR 1.27, 95% CI 1.23-1.31) were associated with increased risk of COVID-19 infection. Insulin treatment was associated with greater COVID-19 infection risk (OR 1.43, 95% CI 1.34-1.52) than treatment with non-insulin drugs (OR 1.26, 95% 1.20-1.33) or no treatment (OR 1.24; 1.18-1.29). The relationship between glycemic control and COVID-19 infection risk was dose-dependent: from an OR of 1.21 (95% CI 1.15-1.26) for hemoglobin A1c (HbA1c) < 7% to an OR of 1.62 (95% CI 1.51-1.75) for HbA1c ≥ 9%. Risk factors for severe COVID-19 were type 1 diabetes (OR 2.87; 95% CI 1.99-4.15), type 2 diabetes (OR 1.80; 95% CI 1.55-2.09), insulin treatment (OR 2.65; 95% CI 2.13-3.28), and HbA1c ≥ 9% (OR 2.61; 95% CI 1.94-3.52). CONCLUSIONS: Diabetes and greater diabetes severity were associated with increased risks of COVID-19 infection and worse COVID-19 outcomes.

Empirical evaluation of internal validation methods for prediction in large-scale clinical data with rare-event outcomes: a case study in suicide risk prediction.

BACKGROUND: There is increasing interest in clinical prediction models for rare outcomes such as suicide, psychiatric hospitalizations, and opioid overdose. Accurate model validation is needed to guide model selection and decisions about whether and how prediction models should be used. Split-sample estimation and validation of clinical prediction models, in which data are divided into training and testing sets, may reduce predictive accuracy and precision of validation. Using all data for estimation and validation increases sample size for both procedures, but validation must account for overfitting, or optimism. Our study compared split-sample and entire-sample methods for estimating and validating a suicide prediction model. METHODS: We compared performance of random forest models estimated in a sample of 9,610,318 mental health visits ("entire-sample") and in a 50% subset ("split-sample") as evaluated in a prospective validation sample of 3,754,137 visits. We assessed optimism of three internal validation approaches: for the split-sample prediction model, validation in the held-out testing set and, for the entire-sample model, cross-validation and bootstrap optimism correction. RESULTS: The split-sample and entire-sample prediction models showed similar prospective performance; the area under the curve, AUC, and 95% confidence interval was 0.81 (0.77-0.85) for both. Performance estimates evaluated in the testing set for the split-sample model (AUC = 0.85 [0.82-0.87]) and via cross-validation for the entire-sample model (AUC = 0.83 [0.81-0.85]) accurately reflected prospective performance. Validation of the entire-sample model with bootstrap optimism correction overestimated prospective performance (AUC = 0.88 [0.86-0.89]). Measures of classification accuracy, including sensitivity and positive predictive value at the 99th, 95th, 90th, and 75th percentiles of the risk score distribution, indicated similar conclusions: bootstrap optimism correction overestimated classification accuracy in the prospective validation set. CONCLUSIONS: While previous literature demonstrated the validity of bootstrap optimism correction for parametric models in small samples, this approach did not accurately validate performance of a rare-event prediction model estimated with random forests in a large clinical dataset. Cross-validation of prediction models estimated with all available data provides accurate independent validation while maximizing sample size.

Variable selection in regression-based estimation of dynamic treatment regimes.

Dynamic treatment regimes (DTRs) consist of a sequence of decision rules, one per stage of intervention, that aim to recommend effective treatments for individual patients according to patient information history. DTRs can be estimated from models which include interactions between treatment and a (typically small) number of covariates which are often chosen a priori. However, with increasingly large and complex data being collected, it can be difficult to know which prognostic factors might be relevant in the treatment rule. Therefore, a more data-driven approach to select these covariates might improve the estimated decision rules and simplify models to make them easier to interpret. We propose a variable selection method for DTR estimation using penalized dynamic weighted least squares. Our method has the strong heredity property, that is, an interaction term can be included in the model only if the corresponding main terms have also been selected. We show our method has both the double robustness property and the oracle property theoretically; and the newly proposed method compares favorably with other variable selection approaches in numerical studies. We further illustrate the proposed method on data from the Sequenced Treatment Alternatives to Relieve Depression study.

Increased COVID-19 Infection Risk Drives Racial and Ethnic Disparities in Severe COVID-19 Outcomes.

COVID-19 inequities have been well-documented. We evaluated whether higher rates of severe COVID-19 in racial and ethnic minority groups were driven by higher infection rates by evaluating if disparities remained when analyses were restricted to people with infection. We conducted a retrospective cohort study of adults insured through Kaiser Permanente (Colorado, Northwest, Washington), follow-up in March-September 2020. Laboratory results and hospitalization diagnosis codes identified individuals with COVID-19. Severe COVID-19 was defined as invasive mechanical ventilation or mortality. Self-reported race and ethnicity, demographics, and medical comorbidities were extracted from health records. Modified Poisson regression estimated adjusted relative risks (aRRs) of severe COVID-19 in full cohort and among individuals with infection. Our cohort included 1,052,774 individuals, representing diverse racial and ethnic minority groups (e.g., 68,887 Asian, 41,243 Black/African American, 93,580 Hispanic or Latino/a individuals). Among 7,399 infections, 442 individuals experienced severe COVID-19. In the full cohort, severe COVID-19 aRRs for Asian, Black/African American, and Hispanic individuals were 2.09 (95% CI: 1.36, 3.21), 2.02 (1.39, 2.93), and 2.09 (1.57, 2.78), respectively, compared to non-Hispanic Whites. In analyses restricted to individuals with COVID-19, all aRRs were near 1, except among Asian Americans (aRR 1.82 [1.23, 2.68]). These results indicate increased incidence of severe COVID-19 among Black/African American and Hispanic individuals is due to higher infection rates, not increased susceptibility to progression. COVID-19 disparities most likely result from social, not biological, factors. Future work should explore reasons for increased severe COVID-19 risk among Asian Americans. Our findings highlight the importance of equity in vaccine distribution.

Identifying hypertensive disorders of pregnancy, a comparison of two epidemiologic definitions.

Introduction: Studies of hypertension in pregnancy that use electronic health care data generally identify hypertension using hospital diagnosis codes alone. We sought to compare results from this approach to an approach that included diagnosis codes, antihypertensive medications and blood pressure (BP) values. Materials and methods: We conducted a retrospective cohort study of 1,45,739 pregnancies from 2009 to 2014 within an integrated healthcare system. Hypertensive pregnancies were identified using the "BP-Inclusive Definition" if at least one of three criteria were met: (1) two elevated outpatient BPs, (2) antihypertensive medication fill plus an outpatient hypertension diagnosis, or (3) hospital discharge diagnosis for preeclampsia or eclampsia. The "Traditional Definition" considered only delivery hospitalization discharge diagnoses. Outcome event analyses compared rates of preterm delivery and small for gestational age (SGA) between the two definitions. Results: The BP-Inclusive Definition identified 14,225 (9.8%) hypertensive pregnancies while the Traditional Definition identified 13,637 (9.4%); 10,809 women met both definitions. Preterm delivery occurred in 20.9% of BP-Inclusive Definition pregnancies, 21.8% of Traditional Definition pregnancies and 6.6% of non-hypertensive pregnancies; for SGA the numbers were 15.6, 16.3, and 8.6%, respectively (p < 0.001 for all events compared to non-hypertensive pregnancies). Analyses in women meeting only one hypertension definition (21-24% of positive cases) found much lower rates of both preterm delivery and SGA. Conclusion: Prevalence of hypertension in pregnancy was similar between the two study definitions. However, a substantial number of women met only one of the study definitions. Women who met only one of the hypertension definitions had much lower rates of adverse neonatal events than women meeting both definitions.

Machine Learning Prediction of Suicide Risk Does Not Identify Patients Without Traditional Risk Factors.

Objective: To determine whether predictions of suicide risk from machine learning models identify unexpected patients or patients without medical record documentation of traditional risk factors.Methods: The study sample included 27,091,382 outpatient mental health (MH) specialty or general medical visits with a MH diagnosis for patients aged 11 years or older from January 1, 2009, to September 30, 2017. We used predicted risk scores of suicide attempt and suicide death, separately, within 90 days of visits to classify visits into risk score percentile strata. For each stratum, we calculated counts and percentages of visits with traditional risk factors, including prior self-harm diagnoses and emergency department visits or hospitalizations with MH diagnoses, in the last 3, 12, and 60 months.Results: Risk-factor percentages increased with predicted risk scores. Among MH specialty visits, 66%, 88%, and 99% of visits with suicide attempt risk scores in the top 3 strata (respectively, 90th-95th, 95th-98th, and ≥ 98th percentiles) and 60%, 77%, and 93% of visits with suicide risk scores in the top 3 strata represented patients who had at least one traditional risk factor documented in the prior 12 months. Among general medical visits, 52%, 66%, and 90% of visits with suicide attempt risk scores in the top 3 strata and 45%, 66%, and 79% of visits with suicide risk scores in the top 3 strata represented patients who had a history of traditional risk factors in the last 12 months.Conclusions: Suicide risk alerts based on these machine learning models coincide with patients traditionally thought of as high-risk at their high-risk visits.