Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction.

Suicide risk prediction models can identify individuals for targeted intervention. Discussions of transparency, explainability, and transportability in machine learning presume complex prediction models with many variables outperform simpler models. We compared random forest, artificial neural network, and ensemble models with 1500 temporally defined predictors to logistic regression models. Data from 25,800,888 mental health visits made by 3,081,420 individuals in 7 health systems were used to train and evaluate suicidal behavior prediction models. Model performance was compared across several measures. All models performed well (area under the receiver operating curve [AUC]: 0.794-0.858). Ensemble models performed best, but improvements over a regression model with 100 predictors were minimal (AUC improvements: 0.006-0.020). Results are consistent across performance metrics and subgroups defined by race, ethnicity, and sex. Our results suggest simpler parametric models, which are easier to implement as part of routine clinical practice, perform comparably to more complex machine learning methods.

Opioid-related variables did not improve suicide risk prediction models in samples with mental health diagnoses.

Background: Suicide risk prediction models derived from machine learning of electronic health records and insurance claims are an innovation in suicide prevention. Some models do not include opioid-related variables despite the relationship between opioids and suicide. This study evaluated whether inclusion of opioid-related variables improved suicide risk prediction models developed by the Mental Health Research Network. Methods: Approximately 630 opioid-related variables and interactions terms were introduced into existing suicide prediction models run in datasets of patient visits in mental health care (n = 27,755,401 visits) or primary care when a mental health diagnosis was given (n = 19,340,461 visits). Training and validation datasets were created. LASSO regression with 10-fold validation identified variables to be added to the models. Results: The new models predicting suicide attempts and suicide deaths in the mental health specialty visit sample performed as well as the existing models (new C-statistic for attempts model = 0.855, CI: 0.853-0.857 versus original C-statistic = 0.851, CI 0.848-0.853; death model = 0.868, CI: 0.856-0.879 versus 0.861, CI 0.848-0.875). The new model for suicide death in the primary care sample improved (0.855, CI: 0.837-0.874 versus 0.833, CI 0.813-0.853) while performance of the new model for suicide attempt in that sample degraded (0.843, CI: 0.839-0.847 versus 0.853, CI 0.849-0.857). Limitations: Analyses did not include patients without recent care, data did not include illicit opioid use or unrecognized opioid use disorder. Conclusions: Among patients with mental health diagnoses, inclusion of opioid-related variables did not improve prediction of suicide risk beyond mental health predictors.

A value proposition for pragmatic clinical trials.

The authors examine the origin, benefits, and challenges of pragmatic clinical trials to assess the ultimate value of this research design.

Suicide Deaths Before and During the Coronavirus Disease 2019 Pandemic: An Interrupted Time-series Study.

INTRODUCTION: With stressors that are often associated with suicide increasing during the coronavirus disease 2019 (COVID-19) pandemic, there has been concern that suicide mortality rates may also be increasing. Our objective was to determine whether suicide mortality rates increased during the COVID-19 pandemic. METHODS: We conducted an interrupted time-series study using data from January 2019 through December 2020 from 2 large integrated health care systems. The population at risk included all patients or individuals enrolled in a health plan at HealthPartners in Minnesota or Henry Ford Health System in Michigan. The primary outcome was change in suicide mortality rates, expressed as annualized crude rates of suicide death per 100,000 people in 10 months following the start of the pandemic in March 2020 compared with the 14 months prior. RESULTS: There were 6,434,675 people at risk in the sample, with 55% women and a diverse sample across ages, race/ethnicity, and insurance type. From January 2019 through February 2020, there was a slow increase in the suicide mortality rate, with rates then decreasing by 0.45 per 100,000 people per month from March 2020 through December 2020 (SE=0.19, P=0.03). CONCLUSIONS: Overall suicide mortality rates did not increase with the pandemic, and in fact slightly declined from March to December 2020. Our findings should be confirmed across other settings and, when available, using final adjudicated state mortality data.

Connecting Research and Practice: Implementation of Suicide Prevention Strategies in Learning Health Care Systems.

The health care systems affiliated with the Mental Health Research Network strive to be learning health care systems that identify and address evidence gaps of importance to clinicians, patients, and funders. This column describes how research guides clinical care and clinical care guides research in the area of suicide prevention as well as some of the challenges of conducting embedded research.

"What Will Happen If I Say Yes?" Perspectives on a Standardized Firearm Access Question Among Adults With Depressive Symptoms.

OBJECTIVE: Addressing firearm access is recommended when patients are identified as being at risk of suicide. However, the practice of assessing firearm access is controversial, and no national guidelines exist to inform practice. This study qualitatively explored patient perspectives on a routine question about firearm access to optimize the patient centeredness of this practice in the context of suicide risk. METHODS: Electronic health record data were used to identify primary care patients reporting depressive symptoms, including suicidal thoughts, within 2 weeks of sampling. Participants completed a semistructured telephone interview (recorded and transcribed), which focused broadly on the experience of being screened for suicidality and included specific questions to elicit beliefs and opinions about being asked a standard firearm access question. Directive (deductive) and conventional (inductive) content analysis was used to analyze responses to the portion of the interview focused on firearm assessment and disclosure. RESULTS: Thirty-seven patients in Washington State ages 20-95 completed the qualitative interview by phone. Organizing themes included apprehensions about disclosing access to firearms related to privacy, autonomy, and firearm ownership rights; perceptions regarding relevance of the firearm question, informed by experiences with suicidality and common beliefs and misconceptions about the inevitability of suicide; and suggestions for connecting questions about firearms and other lethal means to suicide risk. CONCLUSIONS: Clarifying the purpose and use of routine firearm access assessment, contextualizing firearm questions within injury prevention broadly, and addressing misconceptions about suicide prevention may help encourage disclosure of firearm access and increase the patient centeredness of this practice.

Provider Contributions to Disparities in Mental Health Care.

OBJECTIVE: Disparities in diagnosis of mental health problems and in access to treatment among racial-ethnic groups are apparent across different behavioral conditions, particularly in the quality of treatment for depression. This study aimed to determine how much disparities differ across providers. METHODS: Bayesian mixed-effects models were used to estimate whether disparities in patient adherence to antidepressant medication (N=331,776) or psychotherapy (N=275,095) were associated with specific providers. Models also tested whether providers who achieved greater adherence to treatment, on average, among non-Hispanic white patients than among patients from racial-ethnic minority groups attained lower disparities and whether the percentage of patients from racial-ethnic minority groups in a provider caseload was associated with disparities. RESULTS: Disparities in adherence to both antidepressant medication and psychotherapy were associated with the provider. Provider performance with non-Hispanic white patients was negatively correlated with provider-specific disparities in adherence to psychotherapy but not to antidepressants. A higher proportion of patients from racial-ethnic minority groups in a provider's caseload was associated with lower adherence among non-Hispanic white patients, lower disparities in adherence to psychotherapy, and greater disparities in adherence to antidepressant medication. CONCLUSIONS: Adherence to depression treatment among a provider's patients from racial-ethnic minority groups was related to adherence among that provider's non-Hispanic white patients, but evidence also suggested provider-specific disparities. Efforts among providers to decrease disparities might focus on improving the general skill of providers who treat more patients from racial-ethnic minority groups as well as offering culturally based training to providers with notable disparities.

Association Between Patterns of Alcohol Use and Short-Term Risk of Suicide Attempt Among Patients With and Without Reported Suicidal Ideation.

OBJECTIVE: To evaluate the association between patterns of alcohol use and short-term risk of suicide attempt among patients with and without reported suicidal ideation. METHODS: Kaiser Permanente Washington electronic health record data were used to identify mental health visits (1/1/2010-6/30/2015) with documented assessments for unhealthy alcohol use (AUDIT-C) and suicidal ideation (PHQ-9 ninth question). Logistic regression fit using generalized estimating equations were used to conduct visit-level analyses, accounting for correlation between individuals' assessments. Separate models evaluated the association between (1) level of alcohol consumption and (2) frequency of heavy episodic drinking (HED), in combination with suicidal ideation (any vs none), with suicide attempt within 90 days following each visit. Primary models adjusted for age, gender, race/ethnicity and visit year. RESULTS: Of 59,705 visits (43,706 unique patients), 372 (0.62%) were followed by a suicide attempt within 90 days. The risk of suicide attempt was significantly higher for patients reporting suicidal ideation across all levels of alcohol consumption compared to patients reporting low-level alcohol use and no suicidal ideation, particularly high-level use (OR 9.77, 95% CI, 6.23-15.34). Similarly, risk of suicide attempt was higher for patients reporting suicidal ideation across all levels of HED relative to those reporting no HED or suicidal ideation, particularly HED monthly or more (OR 6.80, 95% CI 4.77-9.72). Among patients reporting no suicidal ideation, no associations were observed. CONCLUSIONS: Findings underscore the potential value of offering alcohol-related care to patient reporting suicidal ideation. Additional strategies are needed to identify suicide risk among those reporting no suicidal ideation.

Chronic pain diagnoses and opioid dispensings among insured individuals with serious mental illness.

BACKGROUND: Individuals with major depressive disorder (MDD) and bipolar disorder (BD) have particularly high rates of chronic non-cancer pain (CNCP) and are also more likely to receive prescription opioids for their pain. However, there have been no known studies published to date that have examined opioid treatment patterns among individuals with schizophrenia. METHODS: Using electronic medical record data across 13 Mental Health Research Network sites, individuals with diagnoses of MDD (N = 65,750), BD (N = 38,117) or schizophrenia or schizoaffective disorder (N = 12,916) were identified and matched on age, sex and Medicare status to controls with no documented mental illness. CNCP diagnoses and prescription opioid medication dispensings were extracted for the matched samples. Multivariate analyses were conducted to evaluate (1) the odds of receiving a pain-related diagnosis and (2) the odds of receiving opioids, by separate mental illness diagnosis category compared with matched controls, controlling for age, sex, Medicare status, race/ethnicity, income, medical comorbidities, healthcare utilization and chronic pain diagnoses. RESULTS: Multivariable models indicated that having a MDD (OR = 1.90; 95% CI = 1.85-1.95) or BD (OR = 1.71; 95% CI = 1.66-1.77) diagnosis was associated with increased odds of a CNCP diagnosis after controlling for age, sex, race, income, medical comorbidities and healthcare utilization. By contrast, having a schizophrenia diagnosis was associated with decreased odds of receiving a chronic pain diagnosis (OR = 0.86; 95% CI = 0.82-0.90). Having a MDD (OR = 2.59; 95% CI = 2.44-2.75) or BD (OR = 2.12; 95% CI = 1.97-2.28) diagnosis was associated with increased odds of receiving chronic opioid medications, even after controlling for age, sex, race, income, medical comorbidities, healthcare utilization and chronic pain diagnosis; having a schizophrenia diagnosis was not associated with receiving chronic opioid medications. CONCLUSIONS: Individuals with serious mental illness, who are most at risk for developing opioid-related problems, continue to be prescribed opioids more often than their peers without mental illness. Mental health clinicians may be particularly well-suited to lead pain assessment and management efforts for these patients. Future research is needed to evaluate the effectiveness of involving mental health clinicians in these efforts.

Short-term risk of suicide attempt associated with patterns of patient-reported alcohol use determined by routine AUDIT-C among adults receiving mental healthcare.

OBJECTIVE: To evaluate the association between alcohol use routinely reported during outpatient mental healthcare visits and short-term risk of subsequent suicide attempt. METHODS: Using a longitudinal retrospective-cohort design, electronic health records identified adult outpatient visits to a mental health provider (1/1/2010-6/30/2015) at Kaiser Permanente Washington with a documented Alcohol Use Disorders Identification Test-Consumption [AUDIT-C]. Suicide attempts within 90 days of AUDIT-C documentation were defined using death certificate cause-of-death and diagnosis codes (non-lethal). Visit-level analyses used generalized estimating equations to account for correlation between multiple AUDIT-Cs for individuals. Separate models evaluated the association between (1) level of consumption and (2) frequency of heavy drinking episodes and suicide attempts, adjusted for visit year, demographics, depressive symptom, and suicidal ideation. RESULTS: Of 59,382 patient visits, 0.62% (N = 371) were followed by a suicide attempt within 90 days. Patients reporting high-level alcohol use were 1.77 times (95% CI, 1.22-2.57) more likely to attempt suicide than those reporting low-level use. Patients reporting daily or almost daily heavy drinking episodes were 2.33 times (95% CI, 1.38-3.93) more likely to attempt suicide than those reporting none. CONCLUSIONS AND RELEVANCE: The AUDIT-C is a valuable tool for assessing patterns of patient-reported alcohol use associated with subsequent suicide attempt.

If You Listen, I Will Talk: the Experience of Being Asked About Suicidality During Routine Primary Care.

BACKGROUND: Routine population-based screening for depression is an essential part of evolving health care models integrating care for mental health in primary care. Depression instruments often include questions about suicidal thoughts, but how patients experience these questions in primary care is not known and may have implications for accurate identification of patients at risk. OBJECTIVES: To explore the patient experience of routine population-based depression screening/assessment followed, for some, by suicide risk assessment and discussions with providers. DESIGN: Qualitative, interview-based study. PARTICIPANTS: Thirty-seven patients from Kaiser Permanente Washington who had recently screened positive for depression on the 2-item Patient Health Questionnaire [PHQ] and completed the full PHQ-9. APPROACH: Criterion sampling identified patients who had recently completed the PHQ-9 ninth question which asks about the frequency of thoughts about self-harm. Patients completed semi-structured interviews by phone, which were recorded and transcribed. Directive and conventional content analyses were used to apply knowledge from prior research and elucidate new information from interviews; thematic analysis was used to organize key content overall and across groups based on endorsement of suicide ideation. KEY RESULTS: Four main organizing themes emerged from analyses: (1) Participants believed being asked about suicidality was contextually appropriate and valuable, (2) some participants described a mismatch between their lived experience and the PHQ-9 ninth question, (3) suicidality disclosures involved weighing hope for help against fears of negative consequences, and (4) provider relationships and acts of listening and caring facilitated discussions about suicidality. CONCLUSIONS: All participants believed being asked questions about suicidal thoughts was appropriate, though some who disclosed suicidal thoughts described experiencing stigma and sometimes distanced themselves from suicidality. Direct communication with trusted providers, who listened and expressed empathy, bolstered comfort with disclosure. Future research should consider strategies for reducing stigma and encouraging fearless disclosure among primary care patients experiencing suicidality.

Predicting Suicide Attempts for Racial and Ethnic Groups of Patients During Routine Clinical Care.

The aim of this study was to examine variation in suicide ideation and its relationship to risk of suicide attempt in the subsequent 90 days by race and ethnicity. Participants were adults who completed the Patient Health Questionnaire depression module (PHQ9) during an outpatient encounter between January 10, 2010, and December 12, 2012 (N = 509,945 patients; N = 1,228,308 completed PHQ9). Data came from the Virtual Data Warehouse from four health care systems in the Mental Health Research Network. The sample was majority female (73.7%), primarily 30-64 years old (60.1%), healthy (64.5% comorbidity index = 0), and over half were non-Hispanic White (52.9%). Only Asian patients (OR: 1.31; 95% CI: 1.24, 1.39) had higher odds of reporting suicide ideation when compared to non-Hispanic White people. All racial and ethnic groups had increased risk for suicide attempt with increased frequency of suicide ideation. The PHQ9 item 9 can be used as an indicator of suicide ideation and risk for suicide attempt up to 90 days after the reported ideation in racial and ethnic minority patients during routine clinical care.

Are wishes for death or suicidal ideation symptoms of depression in older adults?

OBJECTIVE: Clinicians may question whether thoughts of being better off dead are normal consequences of aging or symptoms of depression. We examine whether thoughts of suicide are as strongly linked to depression severity in older adults as they are in other age groups. METHODS: Cross-sectional cohort study. Participants included 509,945 outpatients >18 years old from four large integrated healthcare systems in the Mental Health Research Network who completed 1.2 million Patient Health Questionnaires (PHQ) and had data to calculate Charlson Comorbidity Index scores from 2010 through 2012. The PHQ8 estimated depression severity, while suicidal ideation was measured using the 9th item of the PHQ. Data were abstracted from a Virtual Data Warehouse. RESULTS: In older adult patients, suicidal ideation was strongly associated with depression severity. Older adults who had at least moderately-severe depression (PHQ8 ≥15) were 48 times more likely (95% CI: 42.8-53.8) to report suicidal ideation than those with minimal or mild symptoms of depression (PHQ8 <10) after adjustment for all other variables in the model, including medical comorbidity burden. CONCLUSIONS: Depression severity was by far the strongest predictor of suicidal ideation in older adult patients. Older patients with suicidal ideation should be screened for depression.

Understanding Why Patients May Not Report Suicidal Ideation at a Health Care Visit Prior to a Suicide Attempt: A Qualitative Study.

OBJECTIVE: The authors sought to understand why patients may not report suicidal ideation at a health care visit prior to a suicide attempt. METHODS: Electronic health record data from Kaiser Permanente Washington were used to identify patients who reported having no suicidal ideation on question 9 of the nine-item Patient Health Questionnaire and who subsequently made a suicide attempt (≤60 days). Semistructured interviews were audio-recorded, transcribed, and analyzed by using a combination of directed (deductive) and conventional (inductive) content analysis to validate and further explore reasons why patients may not report suicidal ideation prior to a suicide attempt. RESULTS: Of 42 adults sampled, 26 agreed to be interviewed, of whom about half were women (N=15) and a majority was white (N=20), with ages ranging from 18 to 63. Key themes were that patients who attempted suicide after having reported no thoughts of self-harm were either not experiencing suicidal ideation at the time of screening or feared the outcome of disclosure, including stigma, overreaction, and loss of autonomy. An additional theme that emerged from the interviews included reports of heavy episodic drinking at the time of the suicide attempt, particularly when suicide was completely unplanned. Patients also identified important aspects of interactions with health care system providers that may facilitate disclosure about suicidal ideation. CONCLUSIONS: Nonjudgmental listening and expressions of caring without overreaction among providers may help patients overcome fear of reporting suicidal ideation. Screening for heavy episodic drinking may help identify individuals who make unplanned suicide attempts.

Implementing Measurement-Based Care in Behavioral Health: A Review.

Importance: Measurement-based care (MBC) is the systematic evaluation of patient symptoms before or during an encounter to inform behavioral health treatment. Despite MBC's demonstrated ability to enhance usual care by expediting improvements and rapidly detecting patients whose health would otherwise deteriorate, it is underused, with typically less than 20% of behavioral health practitioners integrating it into their practice. This narrative review addresses definitional issues, offers a concrete and evaluable operationalization of MBC fidelity, and summarizes the evidence base and utility of MBC. It also synthesizes the extant literature's characterization of barriers to and strategies for supporting MBC implementation, sustainment, and scale-up. Observations: Barriers to implementing MBC occur at multiple levels: patient (eg, concerns about confidentiality breach), practitioner (eg, beliefs that measures are no better than clinical judgment), organization (eg, no resources for training), and system (eg, competing requirements). Implementation science-the study of methods to integrate evidence-based practices such as MBC into routine care-offers strategies to address barriers. These strategies include using measurement feedback systems, leveraging local champions, forming learning collaboratives, training leadership, improving expert consultation with clinical staff, and generating incentives. Conclusions and Relevance: This narrative review, informed by implementation science, offers a 10-point research agenda to improve the integration of MBC into clinical practice: (1) harmonize terminology and specify MBC's core components; (2) develop criterion standard methods for monitoring fidelity and reporting quality of implementation; (3) develop algorithms for MBC to guide psychotherapy; (4) test putative mechanisms of change, particularly for psychotherapy; (5) develop brief and psychometrically strong measures for use in combination; (6) assess the critical timing of administration needed to optimize patient outcomes; (7) streamline measurement feedback systems to include only key ingredients and enhance electronic health record interoperability; (8) identify discrete strategies to support implementation; (9) make evidence-based policy decisions; and (10) align reimbursement structures.

The impact of race and ethnicity on rates of return to psychotherapy for depression.

BACKGROUND: There are many limitations with the evidence base for the role of race and ethnicity in continuation of psychotherapy for depression. METHODS: The study sample consisted of 242,765 patients ≥ 18 years old from six healthcare systems in the Mental Health Research Network (MHRN) who had a new episode of psychotherapy treatment for depression between 1/1/2010 and 12/31/2013. Data were from electronic medical records and organized in a Virtual Data Warehouse (VDW). The odds of racial and ethnic minority patients returning for a second psychotherapy visit within 45 days of the initial session were examined using multilevel regression. RESULTS: The sample was primarily middle aged (68%, 30-64 years old), female (68.5%), and non-Hispanic white (50.7%), had commercial insurance (81.4%), and a low comorbidity burden (68.8% had no major comorbidities). Return rates within 45 days of the first psychotherapy visit were 47.6%. Compared to their non-Hispanic white counterparts, racial and ethnic minority patients were somewhat less likely to return to psychotherapy for a second visit (adjusted odds ratios [aORs] ranged from 0.80 to 0.90). Healthcare system was a much stronger predictor of return rates (aORs ranged from 0.89 to 5.53), while providers accounted for 21.1% of the variance in return rates. CONCLUSIONS: Provider and healthcare system variation were stronger predictors of patient return to psychotherapy than race and ethnicity. More research is needed to understand why providers and healthcare systems determine psychotherapy return rates for patients of all racial and ethnic groups.

Pragmatic (trial) informatics: a perspective from the NIH Health Care Systems Research Collaboratory.

Pragmatic clinical trials (PCTs) are research investigations embedded in health care settings designed to increase the efficiency of research and its relevance to clinical practice. The Health Care Systems Research Collaboratory, initiated by the National Institutes of Health Common Fund in 2010, is a pioneering cooperative aimed at identifying and overcoming operational challenges to pragmatic research. Drawing from our experience, we present 4 broad categories of informatics-related challenges: (1) using clinical data for research, (2) integrating data from heterogeneous systems, (3) using electronic health records to support intervention delivery or health system change, and (4) assessing and improving data capture to define study populations and outcomes. These challenges impact the validity, reliability, and integrity of PCTs. Achieving the full potential of PCTs and a learning health system will require meaningful partnerships between health system leadership and operations, and federally driven standards and policies to ensure that future electronic health record systems have the flexibility to support research.

Suicidal ideation reported on the PHQ9 and risk of suicidal behavior across age groups.

OBJECTIVE: The Joint Commission recommends all patients be screened for suicide. However, differences in suicide attempt and death rates may affect how well tools predict risk across age groups. Our objective was to determine whether item 9 of the Patient Health Questionnaire (PHQ9) predicts risk for suicide attempts and deaths across age groups. METHODS: PHQ9s completed by adult outpatients treated for mental health conditions in 2010-2012 at four Mental Health Research Network-affiliated healthcare systems were used to measure depression severity and suicidal ideation. Suicide attempts were identified via ICD-9 codes and suicide deaths via ICD-10 codes and state death certificates. RESULTS: In all, 939,268PHQ9s were completed by 297,290 outpatients. Compared to those without, those with nearly daily suicidal ideation were 5-to-8 times more likely to attempt suicide and 3-to-11times more likely to die by suicide within 30 days, and 2-to-4times more likely to attempt suicide and 2-to-6 times more likely to die by suicide within 365 days. The increased risk of suicide death for those with any level of suicidal ideation persisted over two years. The relationships between suicide thoughts and attempts and deaths were similar across age groups. LIMITATIONS: Our sample was limited to outpatients completing a PHQ9 and relied on pre-existing clinical and administrative data. CONCLUSIONS: Suicidal ideation reported on the PHQ9 was a robust predictor of suicide attempts and deaths regardless of age, and this increased risk persisted for two years. Healthcare systems should address both the immediate and sustained risk for suicide for patients of all ages.

Trials without tribulations: Minimizing the burden of pragmatic research on healthcare systems.

Pragmatic clinical trials are increasingly common because they have the potential to yield findings that are directly translatable to real-world healthcare settings. Pragmatic clinical trials need to integrate research into clinical workflow without placing an undue burden on the delivery system. This requires a research partnership between investigators and healthcare system representatives. This paper, organized as a series of case studies drawn from our experience in the NIH Health Care Systems Research Collaboratory, presents guidance from informational interviews of physician-scientists, health services researchers, and delivery system leaders who recently launched pragmatic clinical trials.