Predicting alcohol dependence treatment outcomes: a prospective comparative study of clinical psychologists versus 'trained' machine learning models.

BACKGROUND AND AIMS: Clinical staff are typically poor at predicting alcohol dependence treatment outcomes. Machine learning (ML) offers the potential to model complex clinical data more effectively. This study tested the predictive accuracy of ML algorithms demonstrated to be effective in predicting alcohol dependence outcomes, compared with clinical judgement and traditional linear regression. DESIGN: Prospective study. ML models were trained on 1016 previously treated patients (training-set) who attended a hospital-based alcohol and drug clinic. ML models (n = 27), clinical psychologists (n = 10) and a 'traditional' logistic regression model (n = 1) predicted treatment outcome during the initial treatment session of an alcohol dependence programme. SETTING: A 12-week cognitive behavioural therapy (CBT)-based abstinence programme for alcohol dependence in a hospital-based alcohol and drug clinic in Australia. PARTICIPANTS: Prospective predictions were made for 220 new patients (test-set; 70.91% male, mean age = 35.78 years, standard deviation = 9.19). Sixty-nine (31.36%) patients successfully completed treatment. MEASUREMENTS: Treatment success was the primary outcome variable. The cross-validated training-set accuracy of ML models was used to determine optimal parameters for selecting models for prospective prediction. Accuracy, sensitivity, specificity, area under the receiver operator curve (AUC), Brier score and calibration curves were calculated and compared across predictions. FINDINGS: The mean aggregate accuracy of the ML models (63.06%) was higher than the mean accuracy of psychologist predictions (56.36%). The most accurate ML model achieved 70% accuracy, as did logistic regression. Both were more accurate than psychologists (P < 0.05) and had superior calibration. The high specificity for the selected ML (79%) and logistic regression (90%) meant they were significantly (P < 0.001) more effective than psychologists (50%) at correctly identifying patients whose treatment was unsuccessful. For ML and logistic regression, high specificity came at the expense of sensitivity (26 and 31%, respectively), resulting in poor prediction of successful patients. CONCLUSIONS: Machine learning models and logistic regression appear to be more accurate than psychologists at predicting treatment outcomes in an abstinence programme for alcohol dependence, but sensitivity is low.

Machine learning vs addiction therapists: A pilot study predicting alcohol dependence treatment outcome from patient data in behavior therapy with adjunctive medication.

BACKGROUND AND OBJECTIVES: Clinical staff providing addiction treatment predict patient outcome poorly. Prognoses based on linear statistics are rarely replicated. Addiction is a complex non-linear behavior. Incorporating non-linear models, Machine Learning (ML) has successfully predicted treatment outcome when applied in other areas of medicine. Using identical assessment data across the two groups, this study compares the accuracy of ML models versus clinical staff to predict alcohol dependence treatment outcome in behavior therapy using patient data only. METHODS: Machine learning models (n = 28) were constructed ('trained') using demographic and psychometric assessment data from 780 previously treated patients who had undertaken a 12 week, abstinence-based Cognitive Behavioral Therapy program for alcohol dependence. Independent predictions applying assessment data for an additional 50 consecutive patients were obtained from 10 experienced addiction therapists and the 28 trained ML models. The predictive accuracy of the ML models and the addiction therapists was then compared with further investigation of the 10 best models selected by cross-validated accuracy on the training-set. Variables selected as important for prediction by staff and the most accurate ML model were examined. RESULTS: The most accurate ML model (Fuzzy Unordered Rule Induction Algorithm, 74%) was significantly more accurate than the four least accurate clinical staff (51%-40%). However, the robustness of this finding may be limited by the moderate area under the receiver operator curve (AUC = 0.49). There was no significant difference in mean aggregate predictive accuracy between 10 clinical staff (56.1%) and the 28 best models (58.57%). Addiction therapists favoured demographic and consumption variables compared with the ML model using more questionnaire subscales. CONCLUSIONS: The majority of staff and ML models were not more accurate than suggested by chance. However, the best performing prediction models may provide useful adjunctive information to standard clinically available prognostic data to more effectively target treatment approaches in clinical settings.