Predicting risk of suicidal behavior from insurance claims data vs. linked data from insurance claims and electronic health records.

PURPOSE: Observational studies assessing effects of medical products on suicidal behavior often rely on health record data to account for pre-existing risk. We assess whether high-dimensional models predicting suicide risk using data derived from insurance claims and electronic health records (EHRs) are superior to models using data from insurance claims alone. METHODS: Data were from seven large health systems identified outpatient mental health visits by patients aged 11 or older between 1/1/2009 and 9/30/2017. Data for the 5 years prior to each visit identified potential predictors of suicidal behavior typically available from insurance claims (e.g., mental health diagnoses, procedure codes, medication dispensings) and additional potential predictors available from EHRs (self-reported race and ethnicity, responses to Patient Health Questionnaire or PHQ-9 depression questionnaires). Nonfatal self-harm events following each visit were identified from insurance claims data and fatal self-harm events were identified by linkage to state mortality records. Random forest models predicting nonfatal or fatal self-harm over 90 days following each visit were developed in a 70% random sample of visits and validated in a held-out sample of 30%. Performance of models using linked claims and EHR data was compared to models using claims data only. RESULTS: Among 15 845 047 encounters by 1 574 612 patients, 99 098 (0.6%) were followed by a self-harm event within 90 days. Overall classification performance did not differ between the best-fitting model using all data (area under the receiver operating curve or AUC = 0.846, 95% CI 0.839-0.854) and the best-fitting model limited to data available from insurance claims (AUC = 0.846, 95% CI 0.838-0.853). Competing models showed similar classification performance across a range of cut-points and similar calibration performance across a range of risk strata. Results were similar when the sample was limited to health systems and time periods where PHQ-9 depression questionnaires were recorded more frequently. CONCLUSION: Investigators using health record data to account for pre-existing risk in observational studies of suicidal behavior need not limit that research to databases including linked EHR data.

Predicting Outcomes of Antidepressant Treatment in Community Practice Settings.

OBJECTIVE: The authors examined whether machine-learning models could be used to analyze data from electronic health records (EHRs) to predict patients' responses to antidepressant medications. METHODS: EHR data from a Washington State health system identified patients ages ≥13 years who started an antidepressant medication in 2016 in a community practice setting and had a baseline Patient Health Questionnaire-9 (PHQ-9) score of ≥10 and at least one PHQ-9 score recorded 14-180 days later. Potential predictors of a response to antidepressants were extracted from the EHR and included demographic characteristics, psychiatric and substance use diagnoses, past psychiatric medication use, mental health service use, and past PHQ-9 scores. Random-forest and penalized regression analyses were used to build models predicting follow-up PHQ-9 score and a favorable treatment response (≥50% improvement in score). RESULTS: Among 2,469 patients starting antidepressant medication treatment, the mean±SD baseline PHQ-9 score was 17.3±4.5, and the mean lowest follow-up score was 9.2±5.9. Outcome data were available for 72% of the patients. About 48% of the patients had a favorable treatment response. The best-fitting random-forest models yielded a correlation between predicted and observed follow-up scores of 0.38 (95% CI=0.32-0.45) and an area under the receiver operating characteristic curve for a favorable response of 0.57 (95% CI=0.52-0.61). Results were similar for penalized regression models and for models predicting last PHQ-9 score during follow-up. CONCLUSIONS: Prediction models using EHR data were not accurate enough to inform recommendations for or against starting antidepressant medication. Personalization of depression treatment should instead rely on systematic assessment of early outcomes.

Considerations for Subgroup Analyses in Cluster-Randomized Trials Based on Aggregated Individual-Level Predictors.

In research assessing the effect of an intervention or exposure, a key secondary objective often involves assessing differential effects of this intervention or exposure in subgroups of interest; this is often referred to as assessing effect modification or heterogeneity of treatment effects (HTE). Observed HTE can have important implications for policy, including intervention strategies (e.g., will some patients benefit more from intervention than others?) and prioritizing resources (e.g., to reduce observed health disparities). Analysis of HTE is well understood in studies where the independent unit is an individual. In contrast, in studies where the independent unit is a cluster (e.g., a hospital or school) and a cluster-level outcome is used in the analysis, it is less well understood how to proceed if the HTE analysis of interest involves an individual-level characteristic (e.g., self-reported race) that must be aggregated at the cluster level. Through simulations, we show that only individual-level models have power to detect HTE by individual-level variables; if outcomes must be defined at the cluster level, then there is often low power to detect HTE by the corresponding aggregated variables. We illustrate the challenges inherent to this type of analysis in a study assessing the effect of an intervention on increasing COVID-19 booster vaccination rates at long-term care centers.

The electronic health record Risk of Alzheimer's and Dementia Assessment Rule (eRADAR) Brain Health Trial: Protocol for an embedded, pragmatic clinical trial of a low-cost dementia detection algorithm.

BACKGROUND: About half of people living with dementia have not received a diagnosis, delaying access to treatment, education, and support. We previously developed a tool, eRADAR, which uses information in the electronic health record (EHR) to identify patients who may have undiagnosed dementia. This paper provides the protocol for an embedded, pragmatic clinical trial (ePCT) implementing eRADAR in two healthcare systems to determine whether an intervention using eRADAR increases dementia diagnosis rates and to examine the benefits and harms experienced by patients and other stakeholders. METHODS: We will conduct an ePCT within an integrated healthcare system and replicate it in an urban academic medical center. At primary care clinics serving about 27,000 patients age 65 and above, we will randomize primary care providers (PCPs) to have their patients with high eRADAR scores receive targeted outreach (intervention) or usual care. Intervention patients will be offered a "brain health" assessment visit with a clinical research interventionist mirroring existing roles within the healthcare systems. The interventionist will make follow-up recommendations to PCPs and offer support to newly-diagnosed patients. Patients with high eRADAR scores in both study arms will be followed to identify new diagnoses of dementia in the EHR (primary outcome). Secondary outcomes include healthcare utilization from the EHR and patient, family member and clinician satisfaction assessed through surveys and interviews. CONCLUSION: If this pragmatic trial is successful, the eRADAR tool and intervention could be adopted by other healthcare systems, potentially improving dementia detection, patient care and quality of life.

Characteristics of Adolescents Who Use Secure Messaging on a Health System's Patient Portal.

OBJECTIVES: To determine adolescent characteristics associated with patient portal secure messaging use within a health system. METHODS: This study analyzed monthly data from individuals aged 13 to 17 who met study eligibility criteria from 2019 to 2021. The primary outcome was any secure messages sent from an adolescent's account during each observed month. Unadjusted and adjusted associations between adolescent characteristics and secure messaging use were assessed using generalized estimating equations with log link and binomial variance. RESULTS: Of 667 678 observed months, 50.8% occurred among males who were not transgender, 51.5% among those identifying as non-Hispanic white, and 83.3% among the privately insured. The adjusted relative risks of secure messaging use were significantly higher for individuals with female sex and transgender identities (female sex, not transgender: adjusted relative risk [aRR] 1.41, 95% confidence interval [CI] 1.31-1.52; male sex, transgender: aRR 2.39, CI 1.98-2.90, female sex, transgender: aRR 3.01, 95% CI 2.63-3.46; referent male sex, not transgender), those with prior portal use (aRR 22.06, 95% CI 20.48-23.77; referent no use) and those with a recent preventive care visit (aRR 1.09, 95% CI 1.02-1.16; referent no recent visits). The adjusted relative risks of portal secure messaging use were significantly lower among those with public insurance (aRR 0.58, 95% CI 0.50-0.67; referent private). CONCLUSIONS: Adolescents who sent patient portal secure messages differed from those who did not. Interventions to encourage secure messaging use may require tailoring based on patient characteristics.

Assessing the differential item functioning of PHQ-9 items for diverse racial and ethnic adults with mental health and/or substance use disorder diagnoses: A retrospective cohort study.

BACKGROUND: Improving health equity in depression care and suicide screening requires that measures like the Patient Health Questionnaire 9 (PHQ-9) function similarly for diverse racial and ethnic groups. We evaluated PHQ-9 differential item functioning (DIF) between racial/ethnic groups in a retrospective cohort study of secondary electronic health record (EHR) data from eight healthcare systems. METHODS: The population (n = 755,156) included patients aged 18-64 with mental health and/or substance use disorder (SUD) diagnoses who had a PHQ-9 with no missing item data in the EHR for primary care or mental health visits between 1/1/2009-9/30/2017. We drew two random samples of 1000 from the following racial/ethnic groups originally recorded in EHRs (n = 14,000): Hispanic, and non-Hispanic White, Black, Asian, American Indian/Alaska Native, Native Hawaiian/Other Pacific Islander, multiracial. We assessed DIF using iterative hybrid ordinal logistic regression and item response theory with p < 0.01 and 1000 Monte Carlo simulations, where change in model R2 > 0.01 represented non-negligible (e.g., clinically meaningful) DIF. RESULTS: All PHQ-9 items displayed statistically significant, but negligible (e.g., clinically unmeaningful) DIF between compared groups. The negligible DIF varied between random samples, although six items showed negligible DIF between the same comparison groups in both random samples. LIMITATIONS: Our findings may not generalize to disaggregated racial/ethnic groups or persons without mental health and/or SUD diagnoses. CONCLUSIONS: We found the PHQ-9 had clinically unmeaningful cross-cultural DIF for adult patients with mental health and/or SUD diagnoses. Future research could disaggregate race/ethnicity to discern if within-group identification impacts PHQ-9 DIF.

Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction.

Suicide risk prediction models can identify individuals for targeted intervention. Discussions of transparency, explainability, and transportability in machine learning presume complex prediction models with many variables outperform simpler models. We compared random forest, artificial neural network, and ensemble models with 1500 temporally defined predictors to logistic regression models. Data from 25,800,888 mental health visits made by 3,081,420 individuals in 7 health systems were used to train and evaluate suicidal behavior prediction models. Model performance was compared across several measures. All models performed well (area under the receiver operating curve [AUC]: 0.794-0.858). Ensemble models performed best, but improvements over a regression model with 100 predictors were minimal (AUC improvements: 0.006-0.020). Results are consistent across performance metrics and subgroups defined by race, ethnicity, and sex. Our results suggest simpler parametric models, which are easier to implement as part of routine clinical practice, perform comparably to more complex machine learning methods.

Empirical evaluation of internal validation methods for prediction in large-scale clinical data with rare-event outcomes: a case study in suicide risk prediction.

BACKGROUND: There is increasing interest in clinical prediction models for rare outcomes such as suicide, psychiatric hospitalizations, and opioid overdose. Accurate model validation is needed to guide model selection and decisions about whether and how prediction models should be used. Split-sample estimation and validation of clinical prediction models, in which data are divided into training and testing sets, may reduce predictive accuracy and precision of validation. Using all data for estimation and validation increases sample size for both procedures, but validation must account for overfitting, or optimism. Our study compared split-sample and entire-sample methods for estimating and validating a suicide prediction model. METHODS: We compared performance of random forest models estimated in a sample of 9,610,318 mental health visits ("entire-sample") and in a 50% subset ("split-sample") as evaluated in a prospective validation sample of 3,754,137 visits. We assessed optimism of three internal validation approaches: for the split-sample prediction model, validation in the held-out testing set and, for the entire-sample model, cross-validation and bootstrap optimism correction. RESULTS: The split-sample and entire-sample prediction models showed similar prospective performance; the area under the curve, AUC, and 95% confidence interval was 0.81 (0.77-0.85) for both. Performance estimates evaluated in the testing set for the split-sample model (AUC = 0.85 [0.82-0.87]) and via cross-validation for the entire-sample model (AUC = 0.83 [0.81-0.85]) accurately reflected prospective performance. Validation of the entire-sample model with bootstrap optimism correction overestimated prospective performance (AUC = 0.88 [0.86-0.89]). Measures of classification accuracy, including sensitivity and positive predictive value at the 99th, 95th, 90th, and 75th percentiles of the risk score distribution, indicated similar conclusions: bootstrap optimism correction overestimated classification accuracy in the prospective validation set. CONCLUSIONS: While previous literature demonstrated the validity of bootstrap optimism correction for parametric models in small samples, this approach did not accurately validate performance of a rare-event prediction model estimated with random forests in a large clinical dataset. Cross-validation of prediction models estimated with all available data provides accurate independent validation while maximizing sample size.

External Validation of the eRADAR Risk Score for Detecting Undiagnosed Dementia in Two Real-World Healthcare Systems.

BACKGROUND: Fifty percent of people living with dementia are undiagnosed. The electronic health record (EHR) Risk of Alzheimer's and Dementia Assessment Rule (eRADAR) was developed to identify older adults at risk of having undiagnosed dementia using routinely collected clinical data. OBJECTIVE: To externally validate eRADAR in two real-world healthcare systems, including examining performance over time and by race/ethnicity. DESIGN: Retrospective cohort study PARTICIPANTS: 129,315 members of Kaiser Permanente Washington (KPWA), an integrated health system providing insurance coverage and medical care, and 13,444 primary care patients at University of California San Francisco Health (UCSF), an academic medical system, aged 65 years or older without prior EHR documentation of dementia diagnosis or medication. MAIN MEASURES: Performance of eRADAR scores, calculated annually from EHR data (including vital signs, diagnoses, medications, and utilization in the prior 2 years), for predicting EHR documentation of incident dementia diagnosis within 12 months. KEY RESULTS: A total of 7631 dementia diagnoses were observed at KPWA (11.1 per 1000 person-years) and 216 at UCSF (4.6 per 1000 person-years). The area under the curve was 0.84 (95% confidence interval: 0.84-0.85) at KPWA and 0.79 (0.76-0.82) at UCSF. Using the 90th percentile as the cut point for identifying high-risk patients, sensitivity was 54% (53-56%) at KPWA and 44% (38-51%) at UCSF. Performance was similar over time, including across the transition from International Classification of Diseases, version 9 (ICD-9) to ICD-10 codes, and across racial/ethnic groups (though small samples limited precision in some groups). CONCLUSIONS: eRADAR showed strong external validity for detecting undiagnosed dementia in two health systems with different patient populations and differential availability of external healthcare data for risk calculations. In this study, eRADAR demonstrated generalizability from a research sample to real-world clinical populations, transportability across health systems, robustness to temporal changes in healthcare, and similar performance across larger racial/ethnic groups.

Preoperative Depression Status and 5 Year Metabolic and Bariatric Surgery Outcomes in the PCORnet Bariatric Study Cohort.

OBJECTIVE: To examine whether depression status before metabolic and bariatric surgery (MBS) influenced 5-year weight loss, diabetes, and safety/utilization outcomes in the PCORnet Bariatric Study. SUMMARY OF BACKGROUND DATA: Research on the impact of depression on MBS outcomes is inconsistent with few large, long-term studies. METHODS: Data were extracted from 23 health systems on 36,871 patients who underwent sleeve gastrectomy (SG; n=16,158) or gastric bypass (RYGB; n=20,713) from 2005-2015. Patients with and without a depression diagnosis in the year before MBS were evaluated for % total weight loss (%TWL), diabetes outcomes, and postsurgical safety/utilization (reoperations, revisions, endoscopy, hospitalizations, mortality) at 1, 3, and 5 years after MBS. RESULTS: 27.1% of SG and 33.0% of RYGB patients had preoperative depression, and they had more medical and psychiatric comorbidities than those without depression. At 5 years of follow-up, those with depression, versus those without depression, had slightly less %TWL after RYGB, but not after SG (between group difference = 0.42%TWL, P = 0.04). However, patients with depression had slightly larger HbA1c improvements after RYGB but not after SG (between group difference = - 0.19, P = 0.04). Baseline depression did not moderate diabetes remission or relapse, reoperations, revision, or mortality across operations; however, baseline depression did moderate the risk of endoscopy and repeat hospitalization across RYGB versus SG. CONCLUSIONS: Patients with depression undergoing RYGB and SG had similar weight loss, diabetes, and safety/utilization outcomes to those without depression. The effects of depression were clinically small compared to the choice of operation.

A call to integrate health equity into learning health system research training.

In 2016, the Agency for Healthcare Research and Quality (AHRQ) recommended seven domains for training and mentoring researchers in learning health systems (LHS) science. Health equity was not included as a competency domain. This commentary from scholars in the Consortium for Applied Training to Advance the Learning health system with Scholars/Trainees (CATALyST) K12 program recommends that competency domains be extended to reflect growing demands for evidence on health inequities and interventions to alleviate them. We present real-life case studies from scholars in an LHS research training program that illustrate facilitators, challenges, and potential solutions at the program, funder, and research community-level to receiving training and mentorship in health equity-focused LHS science. We recommend actions in four areas for LHS research training programs: (a) integrate health equity throughout the current LHS domains; (b) develop training and mentoring in health equity; (c) establish program evaluation standards for consideration of health equity; and (d) bring forth relevant, extant expertise from the areas of health disparities research, community-based participatory research, and community-engaged health services research. We emphasize that LHS research must acknowledge and build on the substantial existing contributions, mainly by scholars of color, in the health equity field.

Comparative Safety and Effectiveness of Roux-en-Y Gastric Bypass and Sleeve Gastrectomy for Weight Loss and Type 2 Diabetes Across Race and Ethnicity in the PCORnet Bariatric Study Cohort.

Importance: Bariatric surgery is the most effective treatment for severe obesity; yet it is unclear whether the long-term safety and comparative effectiveness of these operations differ across racial and ethnic groups. Objective: To compare outcomes of Roux-en-Y gastric bypass (RYGB) and sleeve gastrectomy (SG) across racial and ethnic groups in the National Patient-Centered Clinical Research Network (PCORnet) Bariatric Study. Design, Setting, and Participants: This was a retrospective, observational, comparative effectiveness cohort study that comprised 25 health care systems in the PCORnet Bariatric Study. Patients were adults and adolescents aged 12 to 79 years who underwent a primary (first nonrevisional) RYGB or SG operation between January 1, 2005, and September 30, 2015, at participating health systems. Patient race and ethnicity included Black, Hispanic, White, other, and unrecorded. Data were analyzed from July 1, 2021, to January 17, 2022. Exposure: RYGB or SG. Outcomes: Percentage total weight loss (%TWL); type 2 diabetes remission, relapse, and change in hemoglobin A1c (HbA1c) level; and postsurgical safety and utilization outcomes (operations, interventions, revisions/conversions, endoscopy, hospitalizations, mortality, 30-day major adverse events) at 1, 3, and 5 years after surgery. Results: A total of 36 871 patients (mean [SE] age, 45.0 [11.7] years; 29 746 female patients [81%]) were included in the weight analysis. Patients identified with the following race and ethnic categories: 6891 Black (19%), 8756 Hispanic (24%), 19 645 White (53%), 826 other (2%), and 783 unrecorded (2%). Weight loss and mean reductions in HbA1c level were larger for RYGB than SG in all years for Black, Hispanic, and White patients (difference in 5-year weight loss: Black, -7.6%; 95% CI, -8.0 to -7.1; P < .001; Hispanic, -6.2%; 95% CI, -6.6 to -5.9; P < .001; White, -5.9%; 95% CI, -6.3 to -5.7; P < .001; difference in change in year 5 HbA1c level: Black, -0.29; 95% CI, -0.51 to -0.08; P = .009; Hispanic, -0.45; 95% CI, -0.61 to -0.29; P < .001; and White, -0.25; 95% CI, -0.40 to -0.11; P = .001.) The magnitude of these differences was small among racial and ethnic groups (1%-3% of %TWL). Black and Hispanic patients had higher risk of hospitalization when they had RYGB compared with SG (hazard ratio [HR], 1.45; 95% CI, 1.17-1.79; P = .001 and 1.48; 95% CI, 1.22-1.79; P < .001, respectively). Hispanic patients had greater risk of all-cause mortality (HR, 2.41; 95% CI, 1.24-4.70; P = .01) and higher odds of a 30-day major adverse event (odds ratio, 1.92; 95% CI, 1.38-2.68; P < .001) for RYGB compared with SG. There was no interaction between race and ethnicity and operation type for diabetes remission and relapse. Conclusions and Relevance: Variability of the comparative effectiveness of operations for %TWL and HbA1c level across race and ethnicity was clinically small; however, differences in safety and utilization outcomes were clinically and statistically significant for Black and Hispanic patients who had RYGB compared with SG. These findings can inform shared decision-making regarding bariatric operation choice for different racial and ethnic groups of patients.

Machine Learning Prediction of Suicide Risk Does Not Identify Patients Without Traditional Risk Factors.

Objective: To determine whether predictions of suicide risk from machine learning models identify unexpected patients or patients without medical record documentation of traditional risk factors.Methods: The study sample included 27,091,382 outpatient mental health (MH) specialty or general medical visits with a MH diagnosis for patients aged 11 years or older from January 1, 2009, to September 30, 2017. We used predicted risk scores of suicide attempt and suicide death, separately, within 90 days of visits to classify visits into risk score percentile strata. For each stratum, we calculated counts and percentages of visits with traditional risk factors, including prior self-harm diagnoses and emergency department visits or hospitalizations with MH diagnoses, in the last 3, 12, and 60 months.Results: Risk-factor percentages increased with predicted risk scores. Among MH specialty visits, 66%, 88%, and 99% of visits with suicide attempt risk scores in the top 3 strata (respectively, 90th-95th, 95th-98th, and ≥ 98th percentiles) and 60%, 77%, and 93% of visits with suicide risk scores in the top 3 strata represented patients who had at least one traditional risk factor documented in the prior 12 months. Among general medical visits, 52%, 66%, and 90% of visits with suicide attempt risk scores in the top 3 strata and 45%, 66%, and 79% of visits with suicide risk scores in the top 3 strata represented patients who had a history of traditional risk factors in the last 12 months.Conclusions: Suicide risk alerts based on these machine learning models coincide with patients traditionally thought of as high-risk at their high-risk visits.

Accuracy of ICD-10-CM encounter diagnoses from health records for identifying self-harm events.

OBJECTIVE: Assess the accuracy of ICD-10-CM coding of self-harm injuries and poisonings to identify self-harm events. MATERIALS AND METHODS: In 7 integrated health systems, records data identified patients reporting frequent suicidal ideation. Records then identified subsequent ICD-10-CM injury and poisoning codes indicating self-harm as well as selected codes in 3 categories where uncoded self-harm events might be found: injuries and poisonings coded as undetermined intent, those coded accidental, and injuries with no coding of intent. For injury and poisoning encounters with diagnoses in those 4 groups, relevant clinical text was extracted from records and assessed by a blinded panel regarding documentation of self-harm intent. RESULTS: Diagnostic codes selected for review include all codes for self-harm, 43 codes for undetermined intent, 26 codes for accidental intent, and 46 codes for injuries without coding of intent. Clinical text was available for review for 285 events originally coded as self-harm, 85 coded as undetermined intent, 302 coded as accidents, and 438 injury events with no coding of intent. Blinded review of full-text clinical records found documentation of self-harm intent in 254 (89.1%) of those originally coded as self-harm, 24 (28.2%) of those coded as undetermined, 24 (7.9%) of those coded as accidental, and 48 (11.0%) of those without coding of intent. CONCLUSIONS: Among patients at high risk, nearly 90% of injuries and poisonings with ICD-10-CM coding of self-harm have documentation of self-harm intent. Reliance on ICD-10-CM coding of intent to identify self-harm would fail to include a small proportion of true self-harm events.

Pragmatic Randomized Study of Targeted Text Message Reminders to Reduce Missed Clinic Visits.

INTRODUCTION: Missed clinic appointments ("no-shows") waste health system resources, decrease physician availability, and may worsen patient outcomes. Appointment reminders reduce no-shows, though evidence on the optimal number of reminders is limited and sending multiple reminders for every visit is costly. Risk prediction models can be used to target reminders for visits that are likely to be missed. METHODS: We conducted a randomized quality improvement project at Kaiser Permanente Washington among patients with primary care and mental health visits with a high no-show risk comparing the effect of one text message reminder (sent 2 business days prior to the appointment) with 2 text message reminders (sent 2 and 3 days prior) on no-shows and same-day cancellations. We estimated the relative risk (RR) of an additional reminder using G-computation with logistic regression adjusted for no-show risk. RESULTS: Between February 27, 2019 and September 23, 2019, a total of 125,076 primary care visits and 33,593 mental health visits were randomized to either 1 or 2 text message reminders. For primary care visits, an additional text message reduced the chance of no-show by 7% (RR = 0.93, 95% CI: 0.89-0.96) and same-day cancellations by 6% (RR = 0.94, 95% CI: 0.90-0.98). In mental health visits, an additional text message reduced the chance of no-show by 11% (RR = 0.89, 95% CI: 0.86-0.93) but did not impact same-day cancellations (RR = 1.02, 95% CI: 0.96-1.11). We did not find effect modification among subgroups defined by visit or patient characteristics. CONCLUSION: Study findings indicate that using a prediction model to target reminders may reduce no-shows and spend health care resources more efficiently.

Predicting outcomes of psychotherapy for depression with electronic health record data.

Predictive analytics with electronic health record (EHR) data holds promise for improving outcomes of psychiatric care. This study evaluated models for predicting outcomes of psychotherapy for depression in a clinical practice setting. EHR data from two large integrated health systems (Kaiser Permanente Colorado and Washington) included 5,554 new psychotherapy episodes with a baseline Patient Health Questionnaire (PHQ-9) score ≥ 10 and a follow-up PHQ-9 14-180 days after treatment initiation. Baseline predictors included demographics and diagnostic, medication, and encounter history. Prediction models for two outcomes-follow-up PHQ-9 score and treatment response (≥ 50% PHQ-9 reduction)-were trained in a random sample of 70% of episodes and validated in the remaining 30%. Two methods were used for modeling: generalized linear regression models with variable selection and random forests. Sensitivity analyses considered alternate predictor, outcome, and model specifications. Predictions of follow-up PHQ-9 scores poorly estimated observed outcomes (mean squared error = 31 for linear regression, 40 for random forest). Predictions of treatment response had low discrimination (AUC = 0.57 for logistic regression, 0.61 for random forest), low classification accuracy, and poor calibration. Sensitivity analyses showed similar results. We note that prediction model performance may vary for settings with different care or EHR documentation practices. In conclusion, prediction models did not accurately predict depression treatment outcomes despite using rich EHR data and advanced analytic techniques. Health systems should proceed cautiously when considering prediction models for psychiatric outcomes using baseline intake information. Transparent research should be conducted to evaluate performance of any model intended for clinical use.

Evaluating the Cross-Cultural Measurement Invariance of the PHQ-9 between American Indian/Alaska Native Adults and Diverse Racial and Ethnic Groups.

BACKGROUND: The Patient Health Questionnaire-9 (PHQ-9), a self-reported depression screening instrument for measurement-based care (MBC), may have cross-cultural measurement invariance (MI) with a regional group of American Indian/Alaska Native (AI/AN) and non-Hispanic White adults. However, to ensure health equity, research was needed on the cross-cultural MI of the PHQ-9 between other groups of AI/AN peoples and diverse populations. METHODS: We assessed the MI of the one-factor PHQ-9 model and five previously identified two-factor models between non-Hispanic AI/AN adults (ages 18-64) from healthcare systems A (n=1,759) and B (n=2,701) using secondary data and robust maximum likelihood estimation. We then tested either fully or partially invariant models for MI between either combined or separate AI/AN groups, respectively, and Hispanic (n=7,974), White (n=7,974), Asian (n=6,988), Black (n=6,213), and Native Hawaiian/Pacific Islander (n=1,370) adults from healthcare system B. All had mental health or substance use disorder diagnoses and were seen in behavioral health or primary care from 1/1/2009-9/30/2017. RESULTS: The one-factor PHQ-9 model was partially invariant, with two-factor models partially, or in one case fully, invariant between AI/AN groups. The one-factor model and three two-factor models were partially invariant between all seven groups, while a two-factor model was fully invariant and another partially invariant between a combined AI/AN group and other racial and ethnic groups. CONCLUSIONS: Achieving health equity in MBC requires ensuring the cross-cultural validity of measurement tools. Before comparing mean scores, PHQ-9 models should be assessed for individual racial and ethnic group fit for adults with mental health or substance use disorders.

Racial/Ethnic Disparities in the Performance of Prediction Models for Death by Suicide After Mental Health Visits.

Importance: Clinical prediction models estimated with health records data may perpetuate inequities. Objective: To evaluate racial/ethnic differences in the performance of statistical models that predict suicide. Design, Setting, and Participants: In this diagnostic/prognostic study, performed from January 1, 2009, to September 30, 2017, with follow-up through December 31, 2017, all outpatient mental health visits to 7 large integrated health care systems by patients 13 years or older were evaluated. Prediction models were estimated using logistic regression with LASSO variable selection and random forest in a training set that contained all visits from a 50% random sample of patients (6 984 184 visits). Performance was evaluated in the remaining 6 996 386 visits, including visits from White (4 031 135 visits), Hispanic (1 664 166 visits), Black (578 508 visits), Asian (313 011 visits), and American Indian/Alaskan Native (48 025 visits) patients and patients without race/ethnicity recorded (274 702 visits). Data analysis was performed from January 1, 2019, to February 1, 2021. Exposures: Demographic, diagnosis, prescription, and utilization variables and Patient Health Questionnaire 9 responses. Main Outcomes and Measures: Suicide death in the 90 days after a visit. Results: This study included 13 980 570 visits by 1 433 543 patients (64% female; mean [SD] age, 42 [18] years. A total of 768 suicide deaths were observed within 90 days after 3143 visits. Suicide rates were highest for visits by patients with no race/ethnicity recorded (n = 313 visits followed by suicide within 90 days, rate = 5.71 per 10 000 visits), followed by visits by Asian (n = 187 visits followed by suicide within 90 days, rate = 2.99 per 10 000 visits), White (n = 2134 visits followed by suicide within 90 days, rate = 2.65 per 10 000 visits), American Indian/Alaskan Native (n = 21 visits followed by suicide within 90 days, rate = 2.18 per 10 000 visits), Hispanic (n = 392 visits followed by suicide within 90 days, rate = 1.18 per 10 000 visits), and Black (n = 65 visits followed by suicide within 90 days, rate = 0.56 per 10 000 visits) patients. The area under the curve (AUC) and sensitivity of both models were high for White, Hispanic, and Asian patients and poor for Black and American Indian/Alaskan Native patients and patients without race/ethnicity recorded. For example, the AUC for the logistic regression model was 0.828 (95% CI, 0.815-0.840) for White patients compared with 0.640 (95% CI, 0.598-0.681) for patients with unrecorded race/ethnicity and 0.599 (95% CI, 0.513-0.686) for American Indian/Alaskan Native patients. Sensitivity at the 90th percentile was 62.2% (95% CI, 59.2%-65.0%) for White patients compared with 27.5% (95% CI, 21.0%-34.7%) for patients with unrecorded race/ethnicity and 10.0% (95% CI, 0%-23.0%) for Black patients. Results were similar for random forest models, with an AUC of 0.812 (95% CI, 0.800-0.826) for White patients compared with 0.676 (95% CI, 0.638-0.714) for patients with unrecorded race/ethnicity and 0.642 (95% CI, 0.579-0.710) for American Indian/Alaskan Native patients and sensitivities at the 90th percentile of 52.8% (95% CI, 50.0%-55.8%) for White patients, 29.3% (95% CI, 22.8%-36.5%) for patients with unrecorded race/ethnicity, and 6.7% (95% CI, 0%-16.7%) for Black patients. Conclusions and Relevance: These suicide prediction models may provide fewer benefits and more potential harms to American Indian/Alaskan Native or Black patients or those with undrecorded race/ethnicity compared with White, Hispanic, and Asian patients. Improving predictive performance in disadvantaged populations should be prioritized to improve, rather than exacerbate, health disparities.

Reconciling Statistical and Clinicians' Predictions of Suicide Risk.

Statistical models, including those based on electronic health records, can accurately identify patients at high risk for a suicide attempt or death, leading to implementation of risk prediction models for population-based suicide prevention in health systems. However, some have questioned whether statistical predictions can really inform clinical decisions. Appropriately reconciling statistical algorithms with traditional clinician assessment depends on whether predictions from these two methods are competing, complementary, or merely duplicative. In June 2019, the National Institute of Mental Health convened a meeting, "Identifying Research Priorities for Risk Algorithms Applications in Healthcare Settings to Improve Suicide Prevention." Here, participants of this meeting summarize key issues regarding the potential clinical application of suicide prediction models. The authors attempt to clarify the key conceptual and technical differences between traditional risk prediction by clinicians and predictions from statistical models, review the limited evidence regarding both the accuracy of and the concordance between these alternative methods of prediction, present a conceptual framework for understanding agreement and disagreement between statistical and clinician predictions, identify priorities for improving data regarding suicide risk, and propose priority questions for future research. Future suicide risk assessment will likely combine statistical prediction with traditional clinician assessment, but research is needed to determine the optimal combination of these two methods.