Predicting risk of suicidal behavior from insurance claims data vs. linked data from insurance claims and electronic health records.

PURPOSE: Observational studies assessing effects of medical products on suicidal behavior often rely on health record data to account for pre-existing risk. We assess whether high-dimensional models predicting suicide risk using data derived from insurance claims and electronic health records (EHRs) are superior to models using data from insurance claims alone. METHODS: Data were from seven large health systems identified outpatient mental health visits by patients aged 11 or older between 1/1/2009 and 9/30/2017. Data for the 5 years prior to each visit identified potential predictors of suicidal behavior typically available from insurance claims (e.g., mental health diagnoses, procedure codes, medication dispensings) and additional potential predictors available from EHRs (self-reported race and ethnicity, responses to Patient Health Questionnaire or PHQ-9 depression questionnaires). Nonfatal self-harm events following each visit were identified from insurance claims data and fatal self-harm events were identified by linkage to state mortality records. Random forest models predicting nonfatal or fatal self-harm over 90 days following each visit were developed in a 70% random sample of visits and validated in a held-out sample of 30%. Performance of models using linked claims and EHR data was compared to models using claims data only. RESULTS: Among 15 845 047 encounters by 1 574 612 patients, 99 098 (0.6%) were followed by a self-harm event within 90 days. Overall classification performance did not differ between the best-fitting model using all data (area under the receiver operating curve or AUC = 0.846, 95% CI 0.839-0.854) and the best-fitting model limited to data available from insurance claims (AUC = 0.846, 95% CI 0.838-0.853). Competing models showed similar classification performance across a range of cut-points and similar calibration performance across a range of risk strata. Results were similar when the sample was limited to health systems and time periods where PHQ-9 depression questionnaires were recorded more frequently. CONCLUSION: Investigators using health record data to account for pre-existing risk in observational studies of suicidal behavior need not limit that research to databases including linked EHR data.

Preoperative Depression Status and 5 Year Metabolic and Bariatric Surgery Outcomes in the PCORnet Bariatric Study Cohort.

OBJECTIVE: To examine whether depression status before metabolic and bariatric surgery (MBS) influenced 5-year weight loss, diabetes, and safety/utilization outcomes in the PCORnet Bariatric Study. SUMMARY OF BACKGROUND DATA: Research on the impact of depression on MBS outcomes is inconsistent with few large, long-term studies. METHODS: Data were extracted from 23 health systems on 36,871 patients who underwent sleeve gastrectomy (SG; n=16,158) or gastric bypass (RYGB; n=20,713) from 2005-2015. Patients with and without a depression diagnosis in the year before MBS were evaluated for % total weight loss (%TWL), diabetes outcomes, and postsurgical safety/utilization (reoperations, revisions, endoscopy, hospitalizations, mortality) at 1, 3, and 5 years after MBS. RESULTS: 27.1% of SG and 33.0% of RYGB patients had preoperative depression, and they had more medical and psychiatric comorbidities than those without depression. At 5 years of follow-up, those with depression, versus those without depression, had slightly less %TWL after RYGB, but not after SG (between group difference = 0.42%TWL, P = 0.04). However, patients with depression had slightly larger HbA1c improvements after RYGB but not after SG (between group difference = - 0.19, P = 0.04). Baseline depression did not moderate diabetes remission or relapse, reoperations, revision, or mortality across operations; however, baseline depression did moderate the risk of endoscopy and repeat hospitalization across RYGB versus SG. CONCLUSIONS: Patients with depression undergoing RYGB and SG had similar weight loss, diabetes, and safety/utilization outcomes to those without depression. The effects of depression were clinically small compared to the choice of operation.

External Validation of the eRADAR Risk Score for Detecting Undiagnosed Dementia in Two Real-World Healthcare Systems.

BACKGROUND: Fifty percent of people living with dementia are undiagnosed. The electronic health record (EHR) Risk of Alzheimer's and Dementia Assessment Rule (eRADAR) was developed to identify older adults at risk of having undiagnosed dementia using routinely collected clinical data. OBJECTIVE: To externally validate eRADAR in two real-world healthcare systems, including examining performance over time and by race/ethnicity. DESIGN: Retrospective cohort study PARTICIPANTS: 129,315 members of Kaiser Permanente Washington (KPWA), an integrated health system providing insurance coverage and medical care, and 13,444 primary care patients at University of California San Francisco Health (UCSF), an academic medical system, aged 65 years or older without prior EHR documentation of dementia diagnosis or medication. MAIN MEASURES: Performance of eRADAR scores, calculated annually from EHR data (including vital signs, diagnoses, medications, and utilization in the prior 2 years), for predicting EHR documentation of incident dementia diagnosis within 12 months. KEY RESULTS: A total of 7631 dementia diagnoses were observed at KPWA (11.1 per 1000 person-years) and 216 at UCSF (4.6 per 1000 person-years). The area under the curve was 0.84 (95% confidence interval: 0.84-0.85) at KPWA and 0.79 (0.76-0.82) at UCSF. Using the 90th percentile as the cut point for identifying high-risk patients, sensitivity was 54% (53-56%) at KPWA and 44% (38-51%) at UCSF. Performance was similar over time, including across the transition from International Classification of Diseases, version 9 (ICD-9) to ICD-10 codes, and across racial/ethnic groups (though small samples limited precision in some groups). CONCLUSIONS: eRADAR showed strong external validity for detecting undiagnosed dementia in two health systems with different patient populations and differential availability of external healthcare data for risk calculations. In this study, eRADAR demonstrated generalizability from a research sample to real-world clinical populations, transportability across health systems, robustness to temporal changes in healthcare, and similar performance across larger racial/ethnic groups.

Empirical evaluation of internal validation methods for prediction in large-scale clinical data with rare-event outcomes: a case study in suicide risk prediction.

BACKGROUND: There is increasing interest in clinical prediction models for rare outcomes such as suicide, psychiatric hospitalizations, and opioid overdose. Accurate model validation is needed to guide model selection and decisions about whether and how prediction models should be used. Split-sample estimation and validation of clinical prediction models, in which data are divided into training and testing sets, may reduce predictive accuracy and precision of validation. Using all data for estimation and validation increases sample size for both procedures, but validation must account for overfitting, or optimism. Our study compared split-sample and entire-sample methods for estimating and validating a suicide prediction model. METHODS: We compared performance of random forest models estimated in a sample of 9,610,318 mental health visits ("entire-sample") and in a 50% subset ("split-sample") as evaluated in a prospective validation sample of 3,754,137 visits. We assessed optimism of three internal validation approaches: for the split-sample prediction model, validation in the held-out testing set and, for the entire-sample model, cross-validation and bootstrap optimism correction. RESULTS: The split-sample and entire-sample prediction models showed similar prospective performance; the area under the curve, AUC, and 95% confidence interval was 0.81 (0.77-0.85) for both. Performance estimates evaluated in the testing set for the split-sample model (AUC = 0.85 [0.82-0.87]) and via cross-validation for the entire-sample model (AUC = 0.83 [0.81-0.85]) accurately reflected prospective performance. Validation of the entire-sample model with bootstrap optimism correction overestimated prospective performance (AUC = 0.88 [0.86-0.89]). Measures of classification accuracy, including sensitivity and positive predictive value at the 99th, 95th, 90th, and 75th percentiles of the risk score distribution, indicated similar conclusions: bootstrap optimism correction overestimated classification accuracy in the prospective validation set. CONCLUSIONS: While previous literature demonstrated the validity of bootstrap optimism correction for parametric models in small samples, this approach did not accurately validate performance of a rare-event prediction model estimated with random forests in a large clinical dataset. Cross-validation of prediction models estimated with all available data provides accurate independent validation while maximizing sample size.

Considerations for Subgroup Analyses in Cluster-Randomized Trials Based on Aggregated Individual-Level Predictors.

In research assessing the effect of an intervention or exposure, a key secondary objective often involves assessing differential effects of this intervention or exposure in subgroups of interest; this is often referred to as assessing effect modification or heterogeneity of treatment effects (HTE). Observed HTE can have important implications for policy, including intervention strategies (e.g., will some patients benefit more from intervention than others?) and prioritizing resources (e.g., to reduce observed health disparities). Analysis of HTE is well understood in studies where the independent unit is an individual. In contrast, in studies where the independent unit is a cluster (e.g., a hospital or school) and a cluster-level outcome is used in the analysis, it is less well understood how to proceed if the HTE analysis of interest involves an individual-level characteristic (e.g., self-reported race) that must be aggregated at the cluster level. Through simulations, we show that only individual-level models have power to detect HTE by individual-level variables; if outcomes must be defined at the cluster level, then there is often low power to detect HTE by the corresponding aggregated variables. We illustrate the challenges inherent to this type of analysis in a study assessing the effect of an intervention on increasing COVID-19 booster vaccination rates at long-term care centers.

Challenges and Opportunities for Using Big Health Care Data to Advance Medical Science and Public Health.

Methodological advancements in epidemiology, biostatistics, and data science have strengthened the research world's ability to use data captured from electronic health records (EHRs) to address pressing medical questions, but gaps remain. We describe methods investments that are needed to curate EHR data toward research quality and to integrate complementary data sources when EHR data alone are insufficient for research goals. We highlight new methods and directions for improving the integrity of medical evidence generated from pragmatic trials, observational studies, and predictive modeling. We also discuss needed methods contributions to further ease data sharing across multisite EHR data networks. Throughout, we identify opportunities for training and for bolstering collaboration among subject matter experts, methodologists, practicing clinicians, and health system leaders to help ensure that methods problems are identified and resulting advances are translated into mainstream research practice more quickly.

Comparative Effectiveness and Safety of Bariatric Procedures for Weight Loss: A PCORnet Cohort Study.

Background: There has been a dramatic shift in use of bariatric procedures, but little is known about their long-term comparative effectiveness. Objective: To compare weight loss and safety among bariatric procedures. Design: Retrospective observational cohort study, January 2005 to September 2015. (ClinicalTrials.gov: NCT02741674). Setting: 41 health systems in the National Patient-Centered Clinical Research Network. Participants: 65 093 patients aged 20 to 79 years with body mass index (BMI) of 35 kg/m2 or greater who had bariatric procedures. Intervention: 32 208 Roux-en-Y gastric bypass (RYGB), 29 693 sleeve gastrectomy (SG), and 3192 adjustable gastric banding (AGB) procedures. Measurements: Estimated percent total weight loss (TWL) at 1, 3, and 5 years; 30-day rates of major adverse events. Results: Total numbers of eligible patients with weight measures at 1, 3, and 5 years were 44 978 (84%), 20 783 (68%), and 7159 (69%), respectively. Thirty-day rates of major adverse events were 5.0% for RYGB, 2.6% for SG, and 2.9% for AGB. One-year mean TWLs were 31.2% (95% CI, 31.1% to 31.3%) for RYGB, 25.2% (CI, 25.1% to 25.4%) for SG, and 13.7% (CI, 13.3% to 14.0%) for AGB. At 1 year, RYGB patients lost 5.9 (CI, 5.8 to 6.1) percentage points more weight than SG patients and 17.7 (CI, 17.3 to 18.1) percentage points more than AGB patients, and SG patients lost 12.0 (CI, 11.6 to 12.5) percentage points more than AGB patients. Five-year mean TWLs were 25.5% (CI, 25.1% to 25.9%) for RYGB, 18.8% (CI, 18.0% to 19.6%) for SG, and 11.7% (CI, 10.2% to 13.1%) for AGB. Patients with diabetes, those with BMI less than 50 kg/m2, those aged 65 years or older, African American patients, and Hispanic patients lost less weight than patients without those characteristics. Limitation: Potential unobserved confounding due to nonrandomized design; electronic health record databases had missing outcome data. Conclusion: Adults lost more weight with RYGB than with SG or AGB at 1, 3, and 5 years; however, RYGB had the highest 30-day rate of major adverse events. Small subgroup differences in weight loss outcomes were observed. Primary Funding Source: Patient-Centered Outcomes Research Institute.

Estimating efficacy in a randomized trial with product nonadherence: application of multiple methods to a trial of preexposure prophylaxis for HIV prevention.

Antiretroviral preexposure prophylaxis (PrEP) for persons at high risk of human immunodeficiency virus infection is a promising new prevention strategy. Six randomized trials of oral PrEP were recently conducted and demonstrated efficacy estimates ranging from 75% to no effect, with nonadherence likely resulting in attenuated estimates of the protective effect of PrEP. In 1 of these trials, the Partners PrEP Study (Kenya and Uganda, 2008-2011), participants (4,747 serodiscordant heterosexual couples) were randomized to receipt of tenofovir (TDF), coformulated TDF/emtricitabine (FTC), or placebo. Intention-to-treat analyses found efficacy estimates of 67% for TDF and 75% for TDF/FTC. We applied multiple methods to data from that trial to estimate the efficacy of PrEP with high adherence, including principal stratification and inverse-probability-of-censoring (IPC) weights. Results were further from the null when correcting for nonadherence: 1) among the strata with an estimated 100% probability of high adherence (TDF hazard ratio (HR) = 0.19, 95% confidence interval (CI): 0.07, 0.56; TDF/FTC HR = 0.12, 95% CI: 0.03, 0.52); 2) with IPC weights used to approximate a continuously adherent population (TDF HR = 0.18, 95% CI: 0.06, 0.53; TDF/FTC HR = 0.15, 95% CI: 0.04, 0.52); and 3) in per-protocol analysis (TDF HR = 0.18, 95% CI: 0.06, 0.53; TDF/FTC HR = 0.16, 95% CI: 0.05, 0.53). Our results suggest that the efficacy of PrEP with high adherence is over 80%.

Characteristics and fate of orthodontic articles submitted for publication: An exploratory study of the American Journal of Orthodontics and Dentofacial Orthopedics.

INTRODUCTION: In this study, we aimed to give insight into the article review process by investigating the characteristics and the fate of manuscripts submitted to the American Journal of Orthodontics and Dentofacial Orthopedics (AJO-DO). METHODS: The following information was obtained for original articles submitted to the AJO-DO in 2008: (1) for rejected articles: the reasons for rejection and the journal of subsequent publication when applicable; (2) for accepted articles: the number of revisions and the time elapsed to publication; and (3) for all articles: study topic, study design, area of origin, and statistically significant findings. Findings were reported using descriptive statistics, the chi-square test for equality of proportions, and multiple regression where appropriate. Post-hoc pair-wise tests were checked against the Bonferroni correction to account for multiple testing. RESULTS: Of the 440 original articles submitted to AJO-DO in 2008, 116 (26%) were accepted and published an average of 21 months (SD, 5 months) after acceptance. Rejected articles totaled 324 (74%), with 137 (42%) finding subsequent publication an average of 22 months (SD, 11 months) after rejection by the AJO-DO. The top 3 reasons for rejection by the AJO-DO were (1) poor study design (59% of rejected articles), (2) outdated or unoriginal topic (42%), and (3) inappropriate for the AJO-DO's audience (27%). Manuscripts rejected for poor study design had the least success for subsequent publication, whereas those rejected as inappropriate for the AJO-DO had the highest rate of publication elsewhere. Area of origin was significantly associated with acceptance by the AJO-DO, with articles from United States and Canada most likely to be accepted (P < 0.01). Articles from countries with the lowest publication rate in the AJO-DO had the highest publication rate elsewhere. The presence of statistically significant findings was shown to be significantly associated with acceptance by the AJO-DO (P = 0.013) but not with publication elsewhere (P = 0.77). CONCLUSIONS: Rejection by the AJO-DO does not preclude publication elsewhere, although articles rejected for poor study design were least likely to be eventually published. Many publishable articles are rejected by the AJO-DO as inappropriate for its readership, and these were the most likely to find publication elsewhere. Articles with the highest chance of acceptance by the AJO-DO were those from the United States and Canada and those reporting statistically significant results.

Importance of variables from different time frames for predicting self-harm using health system data.

Objective: Self-harm risk prediction models developed using health system data (electronic health records and insurance claims information) often use patient information from up to several years prior to the index visit when the prediction is made. Measurements from some time periods may not be available for all patients. Using the framework of algorithm-agnostic variable importance, we study the predictive potential of variables corresponding to different time horizons prior to the index visit and demonstrate the application of variable importance techniques in the biomedical informatics setting. Materials and Methods: We use variable importance to quantify the potential of recent (up to three months before the index visit) and distant (more than one year before the index visit) patient mental health information for predicting self-harm risk using data from seven health systems. We quantify importance as the decrease in predictiveness when the variable set of interest is excluded from the prediction task. We define predictiveness using discriminative metrics: area under the receiver operating characteristic curve (AUC), sensitivity, and positive predictive value. Results: Mental health predictors corresponding to the three months prior to the index visit show strong signal of importance; in one setting, excluding these variables decreased AUC from 0.85 to 0.77. Predictors corresponding to more distant information were less important. Discussion: Predictors from the months immediately preceding the index visit are highly important. Implementation of self-harm prediction models may be challenging in settings where recent data are not completely available (e.g., due to lags in insurance claims processing) at the time a prediction is made. Conclusion: Clinically derived variables from different time frames exhibit varying levels of importance for predicting self-harm. Variable importance analyses can inform whether and how to implement risk prediction models into clinical practice given real-world data limitations. These analyses be applied more broadly in biomedical informatics research to provide insight into general clinical risk prediction tasks.

Predicting Outcomes of Antidepressant Treatment in Community Practice Settings.

OBJECTIVE: The authors examined whether machine-learning models could be used to analyze data from electronic health records (EHRs) to predict patients' responses to antidepressant medications. METHODS: EHR data from a Washington State health system identified patients ages ≥13 years who started an antidepressant medication in 2016 in a community practice setting and had a baseline Patient Health Questionnaire-9 (PHQ-9) score of ≥10 and at least one PHQ-9 score recorded 14-180 days later. Potential predictors of a response to antidepressants were extracted from the EHR and included demographic characteristics, psychiatric and substance use diagnoses, past psychiatric medication use, mental health service use, and past PHQ-9 scores. Random-forest and penalized regression analyses were used to build models predicting follow-up PHQ-9 score and a favorable treatment response (≥50% improvement in score). RESULTS: Among 2,469 patients starting antidepressant medication treatment, the mean±SD baseline PHQ-9 score was 17.3±4.5, and the mean lowest follow-up score was 9.2±5.9. Outcome data were available for 72% of the patients. About 48% of the patients had a favorable treatment response. The best-fitting random-forest models yielded a correlation between predicted and observed follow-up scores of 0.38 (95% CI=0.32-0.45) and an area under the receiver operating characteristic curve for a favorable response of 0.57 (95% CI=0.52-0.61). Results were similar for penalized regression models and for models predicting last PHQ-9 score during follow-up. CONCLUSIONS: Prediction models using EHR data were not accurate enough to inform recommendations for or against starting antidepressant medication. Personalization of depression treatment should instead rely on systematic assessment of early outcomes.

Characteristics of Adolescents Who Use Secure Messaging on a Health System's Patient Portal.

OBJECTIVES: To determine adolescent characteristics associated with patient portal secure messaging use within a health system. METHODS: This study analyzed monthly data from individuals aged 13 to 17 who met study eligibility criteria from 2019 to 2021. The primary outcome was any secure messages sent from an adolescent's account during each observed month. Unadjusted and adjusted associations between adolescent characteristics and secure messaging use were assessed using generalized estimating equations with log link and binomial variance. RESULTS: Of 667 678 observed months, 50.8% occurred among males who were not transgender, 51.5% among those identifying as non-Hispanic white, and 83.3% among the privately insured. The adjusted relative risks of secure messaging use were significantly higher for individuals with female sex and transgender identities (female sex, not transgender: adjusted relative risk [aRR] 1.41, 95% confidence interval [CI] 1.31-1.52; male sex, transgender: aRR 2.39, CI 1.98-2.90, female sex, transgender: aRR 3.01, 95% CI 2.63-3.46; referent male sex, not transgender), those with prior portal use (aRR 22.06, 95% CI 20.48-23.77; referent no use) and those with a recent preventive care visit (aRR 1.09, 95% CI 1.02-1.16; referent no recent visits). The adjusted relative risks of portal secure messaging use were significantly lower among those with public insurance (aRR 0.58, 95% CI 0.50-0.67; referent private). CONCLUSIONS: Adolescents who sent patient portal secure messages differed from those who did not. Interventions to encourage secure messaging use may require tailoring based on patient characteristics.

Assessing the differential item functioning of PHQ-9 items for diverse racial and ethnic adults with mental health and/or substance use disorder diagnoses: A retrospective cohort study.

BACKGROUND: Improving health equity in depression care and suicide screening requires that measures like the Patient Health Questionnaire 9 (PHQ-9) function similarly for diverse racial and ethnic groups. We evaluated PHQ-9 differential item functioning (DIF) between racial/ethnic groups in a retrospective cohort study of secondary electronic health record (EHR) data from eight healthcare systems. METHODS: The population (n = 755,156) included patients aged 18-64 with mental health and/or substance use disorder (SUD) diagnoses who had a PHQ-9 with no missing item data in the EHR for primary care or mental health visits between 1/1/2009-9/30/2017. We drew two random samples of 1000 from the following racial/ethnic groups originally recorded in EHRs (n = 14,000): Hispanic, and non-Hispanic White, Black, Asian, American Indian/Alaska Native, Native Hawaiian/Other Pacific Islander, multiracial. We assessed DIF using iterative hybrid ordinal logistic regression and item response theory with p < 0.01 and 1000 Monte Carlo simulations, where change in model R2 > 0.01 represented non-negligible (e.g., clinically meaningful) DIF. RESULTS: All PHQ-9 items displayed statistically significant, but negligible (e.g., clinically unmeaningful) DIF between compared groups. The negligible DIF varied between random samples, although six items showed negligible DIF between the same comparison groups in both random samples. LIMITATIONS: Our findings may not generalize to disaggregated racial/ethnic groups or persons without mental health and/or SUD diagnoses. CONCLUSIONS: We found the PHQ-9 had clinically unmeaningful cross-cultural DIF for adult patients with mental health and/or SUD diagnoses. Future research could disaggregate race/ethnicity to discern if within-group identification impacts PHQ-9 DIF.

The electronic health record Risk of Alzheimer's and Dementia Assessment Rule (eRADAR) Brain Health Trial: Protocol for an embedded, pragmatic clinical trial of a low-cost dementia detection algorithm.

BACKGROUND: About half of people living with dementia have not received a diagnosis, delaying access to treatment, education, and support. We previously developed a tool, eRADAR, which uses information in the electronic health record (EHR) to identify patients who may have undiagnosed dementia. This paper provides the protocol for an embedded, pragmatic clinical trial (ePCT) implementing eRADAR in two healthcare systems to determine whether an intervention using eRADAR increases dementia diagnosis rates and to examine the benefits and harms experienced by patients and other stakeholders. METHODS: We will conduct an ePCT within an integrated healthcare system and replicate it in an urban academic medical center. At primary care clinics serving about 27,000 patients age 65 and above, we will randomize primary care providers (PCPs) to have their patients with high eRADAR scores receive targeted outreach (intervention) or usual care. Intervention patients will be offered a "brain health" assessment visit with a clinical research interventionist mirroring existing roles within the healthcare systems. The interventionist will make follow-up recommendations to PCPs and offer support to newly-diagnosed patients. Patients with high eRADAR scores in both study arms will be followed to identify new diagnoses of dementia in the EHR (primary outcome). Secondary outcomes include healthcare utilization from the EHR and patient, family member and clinician satisfaction assessed through surveys and interviews. CONCLUSION: If this pragmatic trial is successful, the eRADAR tool and intervention could be adopted by other healthcare systems, potentially improving dementia detection, patient care and quality of life.

Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction.

Suicide risk prediction models can identify individuals for targeted intervention. Discussions of transparency, explainability, and transportability in machine learning presume complex prediction models with many variables outperform simpler models. We compared random forest, artificial neural network, and ensemble models with 1500 temporally defined predictors to logistic regression models. Data from 25,800,888 mental health visits made by 3,081,420 individuals in 7 health systems were used to train and evaluate suicidal behavior prediction models. Model performance was compared across several measures. All models performed well (area under the receiver operating curve [AUC]: 0.794-0.858). Ensemble models performed best, but improvements over a regression model with 100 predictors were minimal (AUC improvements: 0.006-0.020). Results are consistent across performance metrics and subgroups defined by race, ethnicity, and sex. Our results suggest simpler parametric models, which are easier to implement as part of routine clinical practice, perform comparably to more complex machine learning methods.

A call to integrate health equity into learning health system research training.

In 2016, the Agency for Healthcare Research and Quality (AHRQ) recommended seven domains for training and mentoring researchers in learning health systems (LHS) science. Health equity was not included as a competency domain. This commentary from scholars in the Consortium for Applied Training to Advance the Learning health system with Scholars/Trainees (CATALyST) K12 program recommends that competency domains be extended to reflect growing demands for evidence on health inequities and interventions to alleviate them. We present real-life case studies from scholars in an LHS research training program that illustrate facilitators, challenges, and potential solutions at the program, funder, and research community-level to receiving training and mentorship in health equity-focused LHS science. We recommend actions in four areas for LHS research training programs: (a) integrate health equity throughout the current LHS domains; (b) develop training and mentoring in health equity; (c) establish program evaluation standards for consideration of health equity; and (d) bring forth relevant, extant expertise from the areas of health disparities research, community-based participatory research, and community-engaged health services research. We emphasize that LHS research must acknowledge and build on the substantial existing contributions, mainly by scholars of color, in the health equity field.

Pragmatic Randomized Study of Targeted Text Message Reminders to Reduce Missed Clinic Visits.

INTRODUCTION: Missed clinic appointments ("no-shows") waste health system resources, decrease physician availability, and may worsen patient outcomes. Appointment reminders reduce no-shows, though evidence on the optimal number of reminders is limited and sending multiple reminders for every visit is costly. Risk prediction models can be used to target reminders for visits that are likely to be missed. METHODS: We conducted a randomized quality improvement project at Kaiser Permanente Washington among patients with primary care and mental health visits with a high no-show risk comparing the effect of one text message reminder (sent 2 business days prior to the appointment) with 2 text message reminders (sent 2 and 3 days prior) on no-shows and same-day cancellations. We estimated the relative risk (RR) of an additional reminder using G-computation with logistic regression adjusted for no-show risk. RESULTS: Between February 27, 2019 and September 23, 2019, a total of 125,076 primary care visits and 33,593 mental health visits were randomized to either 1 or 2 text message reminders. For primary care visits, an additional text message reduced the chance of no-show by 7% (RR = 0.93, 95% CI: 0.89-0.96) and same-day cancellations by 6% (RR = 0.94, 95% CI: 0.90-0.98). In mental health visits, an additional text message reduced the chance of no-show by 11% (RR = 0.89, 95% CI: 0.86-0.93) but did not impact same-day cancellations (RR = 1.02, 95% CI: 0.96-1.11). We did not find effect modification among subgroups defined by visit or patient characteristics. CONCLUSION: Study findings indicate that using a prediction model to target reminders may reduce no-shows and spend health care resources more efficiently.

Machine Learning Prediction of Suicide Risk Does Not Identify Patients Without Traditional Risk Factors.

Objective: To determine whether predictions of suicide risk from machine learning models identify unexpected patients or patients without medical record documentation of traditional risk factors.Methods: The study sample included 27,091,382 outpatient mental health (MH) specialty or general medical visits with a MH diagnosis for patients aged 11 years or older from January 1, 2009, to September 30, 2017. We used predicted risk scores of suicide attempt and suicide death, separately, within 90 days of visits to classify visits into risk score percentile strata. For each stratum, we calculated counts and percentages of visits with traditional risk factors, including prior self-harm diagnoses and emergency department visits or hospitalizations with MH diagnoses, in the last 3, 12, and 60 months.Results: Risk-factor percentages increased with predicted risk scores. Among MH specialty visits, 66%, 88%, and 99% of visits with suicide attempt risk scores in the top 3 strata (respectively, 90th-95th, 95th-98th, and ≥ 98th percentiles) and 60%, 77%, and 93% of visits with suicide risk scores in the top 3 strata represented patients who had at least one traditional risk factor documented in the prior 12 months. Among general medical visits, 52%, 66%, and 90% of visits with suicide attempt risk scores in the top 3 strata and 45%, 66%, and 79% of visits with suicide risk scores in the top 3 strata represented patients who had a history of traditional risk factors in the last 12 months.Conclusions: Suicide risk alerts based on these machine learning models coincide with patients traditionally thought of as high-risk at their high-risk visits.

Accuracy of ICD-10-CM encounter diagnoses from health records for identifying self-harm events.

OBJECTIVE: Assess the accuracy of ICD-10-CM coding of self-harm injuries and poisonings to identify self-harm events. MATERIALS AND METHODS: In 7 integrated health systems, records data identified patients reporting frequent suicidal ideation. Records then identified subsequent ICD-10-CM injury and poisoning codes indicating self-harm as well as selected codes in 3 categories where uncoded self-harm events might be found: injuries and poisonings coded as undetermined intent, those coded accidental, and injuries with no coding of intent. For injury and poisoning encounters with diagnoses in those 4 groups, relevant clinical text was extracted from records and assessed by a blinded panel regarding documentation of self-harm intent. RESULTS: Diagnostic codes selected for review include all codes for self-harm, 43 codes for undetermined intent, 26 codes for accidental intent, and 46 codes for injuries without coding of intent. Clinical text was available for review for 285 events originally coded as self-harm, 85 coded as undetermined intent, 302 coded as accidents, and 438 injury events with no coding of intent. Blinded review of full-text clinical records found documentation of self-harm intent in 254 (89.1%) of those originally coded as self-harm, 24 (28.2%) of those coded as undetermined, 24 (7.9%) of those coded as accidental, and 48 (11.0%) of those without coding of intent. CONCLUSIONS: Among patients at high risk, nearly 90% of injuries and poisonings with ICD-10-CM coding of self-harm have documentation of self-harm intent. Reliance on ICD-10-CM coding of intent to identify self-harm would fail to include a small proportion of true self-harm events.