Methodological choices and clinical usefulness for machine learning predictions of outcome in Internet-based cognitive behavioural therapy.

BACKGROUND: While psychological treatments are effective, a substantial portion of patients do not benefit enough. Early identification of those may allow for adaptive treatment strategies and improved outcomes. We aimed to evaluate the clinical usefulness of machine-learning (ML) models predicting outcomes in Internet-based Cognitive Behavioural Therapy, to compare ML-related methodological choices, and guide future use of these. METHODS: Eighty main models were compared. Baseline variables, weekly symptoms, and treatment activity were used to predict treatment outcomes in a dataset of 6695 patients from regular care. RESULTS: We show that the best models use handpicked predictors and impute missing data. No ML algorithm shows clear superiority. They have a mean balanced accuracy of 78.1% at treatment week four, closely matched by regression (77.8%). CONCLUSIONS: ML surpasses the benchmark for clinical usefulness (67%). Advanced and simple models perform equally, indicating a need for more data or smarter methodological designs to confirm advantages of ML.

While there are many therapy treatments that are effective for mental health problems some patients don't benefit enough. Predicting whom might need more help can guide therapists to adjust treatments for better results. Computer methods are increasingly used for predicting the outcome of treatment, but studies vary widely in accuracy and methodology. We examined a variety of models to test performance. Those examined were based on a several factors: what data is chosen, how the data is managed, as well as type of mathematical equations and function used for prediction. When used on ~6500 patients, none of the computer methods tested stood out as the best. Simple models were as accurate as more advanced. Accuracy of prediction of treatment outcome was good enough to inform clinicians' decisions, suggesting they may still be useful tools in mental health care.

Predicting maintenance lithium response for bipolar disorder from electronic health records-a retrospective study.

Background: Optimising maintenance drug treatment selection for people with bipolar disorder is challenging. There is some evidence that clinical and demographic features may predict response to lithium. However, attempts to personalise treatment choice have been limited. Method: We aimed to determine if machine learning methods applied to electronic health records could predict differential response to lithium or olanzapine. From electronic United Kingdom primary care records, we extracted a cohort of individuals prescribed either lithium (19,106 individuals) or olanzapine (12,412) monotherapy. Machine learning models were used to predict successful monotherapy maintenance treatment, using 113 clinical and demographic variables, 8,017 (41.96%) lithium responders and 3,831 (30.87%) olanzapine responders. Results: We found a quantitative structural difference in that lithium maintenance responders were weakly predictable in our holdout sample, consisting of the 5% of patients with the most recent exposure. Age at first diagnosis, age at first treatment and the time between these were the most important variables in all models. Discussion: Even if we failed to predict successful monotherapy olanzapine treatment, and so to definitively separate lithium vs. olanzapine responders, the characterization of the two groups may be used for classification by proxy. This can, in turn, be useful for establishing maintenance therapy. The further exploration of machine learning methods on EHR data for drug treatment selection could in the future play a role for clinical decision support. Signals in the data encourage further experiments with larger datasets to definitively separate lithium vs. olanzapine responders.

Predicting remission after internet-delivered psychotherapy in patients with depression using machine learning and multi-modal data.

This study applied supervised machine learning with multi-modal data to predict remission of major depressive disorder (MDD) after psychotherapy. Genotyped adult patients (n = 894, 65.5% women, age 18-75 years) diagnosed with mild-to-moderate MDD and treated with guided Internet-based Cognitive Behaviour Therapy (ICBT) at the Internet Psychiatry Clinic in Stockholm were included (2008-2016). Predictor types were demographic, clinical, process (e.g., time to complete online questionnaires), and genetic (polygenic risk scores). Outcome was remission status post ICBT (cut-off ≤10 on MADRS-S). Data were split into train (60%) and validation (40%) given ICBT start date. Predictor selection employed human expertise followed by recursive feature elimination. Model derivation was internally validated through cross-validation. The final random forest model was externally validated against a (i) null, (ii) logit, (iii) XGBoost, and (iv) blended meta-ensemble model on the hold-out validation set. Feature selection retained 45 predictors representing all four predictor types. With unseen validation data, the final random forest model proved reasonably accurate at classifying post ICBT remission (Accuracy 0.656 [0.604, 0.705], P vs null model = 0.004; AUC 0.687 [0.631, 0.743]), slightly better vs logit (bootstrap D = 1.730, P = 0.084) but not vs XGBoost (D = 0.463, P = 0.643). Transparency analysis showed model usage of all predictor types at both the group and individual patient level. A new, multi-modal classifier for predicting MDD remission status after ICBT treatment in routine psychiatric care was derived and empirically validated. The multi-modal approach to predicting remission may inform tailored treatment, and deserves further investigation to attain clinical usefulness.

Predicting treatment failure in regular care Internet-Delivered Cognitive Behavior Therapy for depression and anxiety using only weekly symptom measures.

OBJECTIVE: Therapist guided Internet-Delivered Cognitive Behavior Therapy (ICBT) is effective, but as in traditional CBT, not all patients improve, and clinicians generally fail to identify them early enough. We predict treatment failure in 12-week regular care ICBT for Depression, Panic disorder and Social anxiety disorder, using only patients' weekly symptom ratings to identify when the accuracy of predictions exceed 2 benchmarks: (a) chance, and (b) empirically derived clinician preferences for actionable predictions. METHOD: Screening, pretreatment and weekly symptom ratings from 4310 regular care ICBT-patients from the Internet Psychiatry Clinic in Stockholm, Sweden was analyzed in a series of regression models each adding 1 more week of data. Final score was predicted in a holdout test sample, which was then categorized into Success or Failure (failure defined as the absence of both remitter and responder status). Classification analyses with Balanced Accuracy and 95% Confidence intervals was then compared to predefined benchmarks. RESULTS: Benchmark 1 (better than chance) was reached 1 week into all treatments. Social anxiety disorder reached Benchmark 2 (> 65%) at week 5, whereas Depression and Panic Disorder reached it at week 6. CONCLUSIONS: For depression, social anxiety and panic disorder, prediction with only patient-rated symptom scores can detect treatment failure 6 weeks into ICBT, with enough accuracy for a clinician to take action. Early identification of failing treatment attempts may be a viable way to increase the overall success rate of existing psychological treatments by providing extra clinical resources to at-risk patients, within a so-called Adaptive Treatment Strategy. (PsycINFO Database Record (c) 2020 APA, all rights reserved).