Predictive models for clinical decision making: Deep dives in practical machine learning.

The deployment of machine learning for tasks relevant to complementing standard of care and advancing tools for precision health has gained much attention in the clinical community, thus meriting further investigations into its broader use. In an introduction to predictive modelling using machine learning, we conducted a review of the recent literature that explains standard taxonomies, terminology and central concepts to a broad clinical readership. Articles aimed at readers with little or no prior experience of commonly used methods or typical workflows were summarised and key references are highlighted. Continual interdisciplinary developments in data science, biostatistics and epidemiology also motivated us to further discuss emerging topics in predictive and data-driven (hypothesis-less) analytics with machine learning. Through two methodological deep dives using examples from precision psychiatry and outcome prediction after lymphoma, we highlight how the use of, for example, natural language processing can outperform established clinical risk scores and aid dynamic prediction and adaptive care strategies. Such realistic and detailed examples allow for critical analysis of the importance of new technological advances in artificial intelligence for clinical decision-making. New clinical decision support systems can assist in prevention and care by leveraging precision medicine.

Prediction of individuals at high risk of chronic kidney disease during treatment with lithium for bipolar disorder.

BACKGROUND: Lithium is the most effective treatment in bipolar disorder. Its use is limited by concerns about risk of chronic kidney disease (CKD). We aimed to develop a model to predict risk of CKD following lithium treatment initiation, by identifying individuals with a high-risk trajectory of kidney function. METHODS: We used United Kingdom Clinical Practice Research Datalink (CPRD) electronic health records (EHRs) from 2000 to 2018. CPRD Aurum for prediction model development and CPRD Gold for external validation. We used elastic net regularised regression to generate a prediction model from potential features. We performed discrimination and calibration assessments in an external validation data set. We included all patients aged ≥ 16 with bipolar disorder prescribed lithium. To be included patients had to have ≥ 1 year of follow-up before lithium initiation, ≥ 3 estimated glomerular filtration rate (eGFR) measures after lithium initiation (to be able to determine a trajectory) and a normal (≥ 60 mL/min/1.73 m2) eGFR at lithium initiation (baseline). In the Aurum development cohort, 1609 fulfilled these criteria. The Gold external validation cohort included 934 patients. We included 44 potential baseline features in the prediction model, including sociodemographic, mental and physical health and drug treatment characteristics. We compared a full model with the 3-variable 5-year kidney failure risk equation (KFRE) and a 3-variable elastic net model. We used group-based trajectory modelling to identify latent trajectory groups for eGFR. We were interested in the group with deteriorating kidney function (the high-risk group). RESULTS: The high risk of deteriorating eGFR group included 191 (11.87%) of the Aurum cohort and 137 (14.67%) of the Gold cohort. Of these, 168 (87.96%) and 117 (85.40%) respectively developed CKD 3a or more severe during follow-up. The model, developed in Aurum, had a ROC area of 0.879 (95%CI 0.853-0.904) in the Gold external validation data set. At the empirical optimal cut-point defined in the development dataset, the model had a sensitivity of 0.91 (95%CI 0.84-0.97) and a specificity of 0.74 (95% CI 0.67-0.82). However, a 3-variable elastic net model (including only age, sex and baseline eGFR) performed similarly well (ROC area 0.888; 95%CI 0.864-0.912), as did the KFRE (ROC area 0.870; 95%CI 0.841-0.898). CONCLUSIONS: Individuals at high risk of a poor eGFR trajectory can be identified before initiation of lithium treatment by a simple equation including age, sex and baseline eGFR. Risk was increased in individuals who were younger at commencement of lithium, female and had a lower baseline eGFR. We did not identify strong predicters of eGFR decline specific to lithium-treated patients. Notably, lithium duration and toxicity were not associated with high-risk trajectory.

Methodological choices and clinical usefulness for machine learning predictions of outcome in Internet-based cognitive behavioural therapy.

BACKGROUND: While psychological treatments are effective, a substantial portion of patients do not benefit enough. Early identification of those may allow for adaptive treatment strategies and improved outcomes. We aimed to evaluate the clinical usefulness of machine-learning (ML) models predicting outcomes in Internet-based Cognitive Behavioural Therapy, to compare ML-related methodological choices, and guide future use of these. METHODS: Eighty main models were compared. Baseline variables, weekly symptoms, and treatment activity were used to predict treatment outcomes in a dataset of 6695 patients from regular care. RESULTS: We show that the best models use handpicked predictors and impute missing data. No ML algorithm shows clear superiority. They have a mean balanced accuracy of 78.1% at treatment week four, closely matched by regression (77.8%). CONCLUSIONS: ML surpasses the benchmark for clinical usefulness (67%). Advanced and simple models perform equally, indicating a need for more data or smarter methodological designs to confirm advantages of ML.

While there are many therapy treatments that are effective for mental health problems some patients don't benefit enough. Predicting whom might need more help can guide therapists to adjust treatments for better results. Computer methods are increasingly used for predicting the outcome of treatment, but studies vary widely in accuracy and methodology. We examined a variety of models to test performance. Those examined were based on a several factors: what data is chosen, how the data is managed, as well as type of mathematical equations and function used for prediction. When used on ~6500 patients, none of the computer methods tested stood out as the best. Simple models were as accurate as more advanced. Accuracy of prediction of treatment outcome was good enough to inform clinicians' decisions, suggesting they may still be useful tools in mental health care.

Making the most out of timeseries symptom data: A machine learning study on symptom predictions of internet-based CBT.

Objective: Predicting who will not benefit enough from Internet-Based Cognitive Behavioral (ICBT) Therapy early on can assist in better allocation of limited mental health care resources. Repeated measures of symptoms during treatment is the strongest predictor of outcome, and we want to investigate if methods that explicitly account for time-dependency are superior to methods that do not, with data from (a) only two pre-treatment timepoints and (b) the pre-treatment timepoints and three timepoints during initial treatment. Methods: We use 1) commonly used time-independent methods (i.e., Linear Regression and Random Forest models) and 2) time-dependent methods (i.e., multilevel model regression, mixed-effects random forest, and a Long Short-Term Memory model) to predict symptoms during treatment, including the final outcome. This is done with symptom scores from 6436 ICBT patients from regular care, using robust multiple imputation and nested cross-validation methods. Results: The models had a 14 %-12 % root mean squared error (RMSE) in predicting the post-treatment outcome, corresponding to a balanced accuracy of 67-74 %. Time-dependent models did not have higher accuracies. Using data for the initial treatment period (b) instead of only from before treatment (a) increased prediction results by 1.3 % percentage points (12 % to 10.7 %) RMSE and 6 % percentage points BACC (69 % to 75 %). Conclusion: Training prediction models on only symptom scores of the first few weeks is a promising avenue for symptom predictions in treatment, regardless of which model is used. Further research is necessary to better understand the interaction between model complexity, dataset length and width, and the prediction tasks at hand.

Predicting maintenance lithium response for bipolar disorder from electronic health records-a retrospective study.

Background: Optimising maintenance drug treatment selection for people with bipolar disorder is challenging. There is some evidence that clinical and demographic features may predict response to lithium. However, attempts to personalise treatment choice have been limited. Method: We aimed to determine if machine learning methods applied to electronic health records could predict differential response to lithium or olanzapine. From electronic United Kingdom primary care records, we extracted a cohort of individuals prescribed either lithium (19,106 individuals) or olanzapine (12,412) monotherapy. Machine learning models were used to predict successful monotherapy maintenance treatment, using 113 clinical and demographic variables, 8,017 (41.96%) lithium responders and 3,831 (30.87%) olanzapine responders. Results: We found a quantitative structural difference in that lithium maintenance responders were weakly predictable in our holdout sample, consisting of the 5% of patients with the most recent exposure. Age at first diagnosis, age at first treatment and the time between these were the most important variables in all models. Discussion: Even if we failed to predict successful monotherapy olanzapine treatment, and so to definitively separate lithium vs. olanzapine responders, the characterization of the two groups may be used for classification by proxy. This can, in turn, be useful for establishing maintenance therapy. The further exploration of machine learning methods on EHR data for drug treatment selection could in the future play a role for clinical decision support. Signals in the data encourage further experiments with larger datasets to definitively separate lithium vs. olanzapine responders.

Smart variant filtering - A blueprint solution for massively parallel sequencing-based variant analysis.

Massively parallel sequencing helps create new knowledge on genes, variants and their association with disease phenotype. This important technological advancement simultaneously makes clinical decision making, using genomic information for cancer patients, more complex. Currently, identifying actionable pathogenic variants with diagnostic, prognostic, or predictive impact requires substantial manual effort. Objective: The purpose is to design a solution for clinical diagnostics of lymphoma, specifically for systematic variant filtering and interpretation. Methods: A scoping review and demonstrations from specialists serve as a basis for a blueprint of a solution for massively parallel sequencing-based genetic diagnostics. Results: The solution uses machine learning methods to facilitate decision making in the diagnostic process. A validation round of interviews with specialists consolidated the blueprint and anchored it across all relevant expert disciplines. The scoping review identified four components of variant filtering solutions: algorithms and Artificial Intelligence (AI) applications, software, bioinformatics pipelines and variant filtering strategies. The blueprint describes the input, the AI model and the interface for dynamic browsing. Conclusion: An AI-augmented system is designed for predicting pathogenic variants. While such a system can be used to classify identified variants, diagnosticians should still evaluate the classification's accuracy, make corrections when necessary, and ultimately decide which variants are truly pathogenic.

Swedish multimodal cohort of patients with anxiety or depression treated with internet-delivered psychotherapy (MULTI-PSYCH).

PURPOSE: Depression and anxiety afflict millions worldwide causing considerable disability. MULTI-PSYCH is a longitudinal cohort of genotyped and phenotyped individuals with depression or anxiety disorders who have undergone highly structured internet-based cognitive-behaviour therapy (ICBT). The overarching purpose of MULTI-PSYCH is to improve risk stratification, outcome prediction and secondary preventive interventions. MULTI-PSYCH is a precision medicine initiative that combines clinical, genetic and nationwide register data. PARTICIPANTS: MULTI-PSYCH includes 2668 clinically well-characterised adults with major depressive disorder (MDD) (n=1300), social anxiety disorder (n=640) or panic disorder (n=728) assessed before, during and after 12 weeks of ICBT at the internet psychiatry clinic in Stockholm, Sweden. All patients have been blood sampled and genotyped. Clinical and genetic data have been linked to several Swedish registers containing a wide range of variables from patient birth up to 10 years after the end of ICBT. These variable types include perinatal complications, school grades, psychiatric and somatic comorbidity, dispensed medications, medical interventions and diagnoses, healthcare and social benefits, demographics, income and more. Long-term follow-up data will be collected through 2029. FINDINGS TO DATE: Initial uses of MULTI-PSYCH include the discovery of an association between PRS for autism spectrum disorder and response to ICBT, the development of a machine learning model for baseline prediction of remission status after ICBT in MDD and data contributions to genome wide association studies for ICBT outcome. Other projects have been launched or are in the planning phase. FUTURE PLANS: The MULTI-PSYCH cohort provides a unique infrastructure to study not only predictors or short-term treatment outcomes, but also longer term medical and socioeconomic outcomes in patients treated with ICBT for depression or anxiety. MULTI-PSYCH is well positioned for research collaboration.

Data-driven support to decision-making in molecular tumour boards for lymphoma: A design science approach.

Background: The increasing amount of molecular data and knowledge about genomic alterations from next-generation sequencing processes together allow for a greater understanding of individual patients, thereby advancing precision medicine. Molecular tumour boards feature multidisciplinary teams of clinical experts who meet to discuss complex individual cancer cases. Preparing the meetings is a manual and time-consuming process. Purpose: To design a clinical decision support system to improve the multimodal data interpretation in molecular tumour board meetings for lymphoma patients at Karolinska University Hospital, Stockholm, Sweden. We investigated user needs and system requirements, explored the employment of artificial intelligence, and evaluated the proposed design with primary stakeholders. Methods: Design science methodology was used to form and evaluate the proposed artefact. Requirements elicitation was done through a scoping review followed by five semi-structured interviews. We used UML Use Case diagrams to model user interaction and UML Activity diagrams to inform the proposed flow of control in the system. Additionally, we modelled the current and future workflow for MTB meetings and its proposed machine learning pipeline. Interactive sessions with end-users validated the initial requirements based on a fictive patient scenario which helped further refine the system. Results: The analysis showed that an interactive secure Web-based information system supporting the preparation of the meeting, multidisciplinary discussions, and clinical decision-making could address the identified requirements. Integrating artificial intelligence via continual learning and multimodal data fusion were identified as crucial elements that could provide accurate diagnosis and treatment recommendations. Impact: Our work is of methodological importance in that using artificial intelligence for molecular tumour boards is novel. We provide a consolidated proof-of-concept system that could support the end-to-end clinical decision-making process and positively and immediately impact patients. Conclusion: Augmenting a digital decision support system for molecular tumour boards with retrospective patient material is promising. This generates realistic and constructive material for human learning, and also digital data for continual learning by data-driven artificial intelligence approaches. The latter makes the future system adaptable to human bias, improving adequacy and decision quality over time and over tasks, while building and maintaining a digital log.

A genetically informed prediction model for suicidal and aggressive behaviour in teens.

Suicidal and aggressive behaviours cause significant personal and societal burden. As risk factors associated with these behaviours frequently overlap, combined approaches in predicting the behaviours may be useful in identifying those at risk for either. The current study aimed to create a model that predicted if individuals will exhibit suicidal behaviour, aggressive behaviour, both, or neither in late adolescence. A sample of 5,974 twins from the Child and Adolescent Twin Study in Sweden (CATSS) was broken down into a training (80%), tune (10%) and test (10%) set. The Netherlands Twin Register (NTR; N = 2702) was used for external validation. Our longitudinal data featured genetic, environmental, and psychosocial predictors derived from parental and self-report data. A stacked ensemble model was created which contained a gradient boosted machine, random forest, elastic net, and neural network. Model performance was transferable between CATSS and NTR (macro area under the receiver operating characteristic curve (AUC) [95% CI] AUCCATSS(test set) = 0.709 (0.671-0.747); AUCNTR = 0.685 (0.656-0.715), suggesting model generalisability across Northern Europe. The notable exception is suicidal behaviours in the NTR, which was no better than chance. The 25 highest scoring variable importance scores for the gradient boosted machines and random forest models included self-reported psychiatric symptoms in mid-adolescence, sex, and polygenic scores for psychiatric traits. The model's performance is comparable to current prediction models that use clinical interviews and is not yet suitable for clinical use. Moreover, genetic variables may have a role to play in predictive models of adolescent psychopathology.

Predicting remission after internet-delivered psychotherapy in patients with depression using machine learning and multi-modal data.

This study applied supervised machine learning with multi-modal data to predict remission of major depressive disorder (MDD) after psychotherapy. Genotyped adult patients (n = 894, 65.5% women, age 18-75 years) diagnosed with mild-to-moderate MDD and treated with guided Internet-based Cognitive Behaviour Therapy (ICBT) at the Internet Psychiatry Clinic in Stockholm were included (2008-2016). Predictor types were demographic, clinical, process (e.g., time to complete online questionnaires), and genetic (polygenic risk scores). Outcome was remission status post ICBT (cut-off ≤10 on MADRS-S). Data were split into train (60%) and validation (40%) given ICBT start date. Predictor selection employed human expertise followed by recursive feature elimination. Model derivation was internally validated through cross-validation. The final random forest model was externally validated against a (i) null, (ii) logit, (iii) XGBoost, and (iv) blended meta-ensemble model on the hold-out validation set. Feature selection retained 45 predictors representing all four predictor types. With unseen validation data, the final random forest model proved reasonably accurate at classifying post ICBT remission (Accuracy 0.656 [0.604, 0.705], P vs null model = 0.004; AUC 0.687 [0.631, 0.743]), slightly better vs logit (bootstrap D = 1.730, P = 0.084) but not vs XGBoost (D = 0.463, P = 0.643). Transparency analysis showed model usage of all predictor types at both the group and individual patient level. A new, multi-modal classifier for predicting MDD remission status after ICBT treatment in routine psychiatric care was derived and empirically validated. The multi-modal approach to predicting remission may inform tailored treatment, and deserves further investigation to attain clinical usefulness.

Stockholm score of lesion detection on computed tomography following mild traumatic brain injury (SELECT-TBI): study protocol for a multicentre, retrospective, observational cohort study.

INTRODUCTION: Mild traumatic brain injury (mTBI) is one of the most common reasons for emergency department (ED) visits. A portion of patients with mTBI will develop an intracranial lesion that might require medical or surgical intervention. In these patients, swift diagnosis and management is paramount. Several guidelines have been developed to help direct patients with mTBI for head CT scanning, but they lack specificity, do not consider the interactions between risk factors and do not provide an individualised estimate of intracranial lesion risk. The aim of this study is to create a model that estimates individualised intracranial lesion risks in patients with mTBI who present to the ED. METHODS AND ANALYSIS: This will be a retrospective cohort study conducted at ED hospitals in Stockholm, Sweden. Eligible patients are adults (≥15 years) with mTBI who presented to the ED within 24 hours of injury and performed a CT scan. The primary outcome will be a traumatic lesion on head CT. The secondary outcomes will be any clinically significant lesion, defined as an intracranial finding that led to neurosurgical intervention, hospital admission ≥48 hours due to TBI or death due to TBI. Machine-learning models will be applied to create scores predicting the primary and secondary outcomes. An estimated 20 000 patients will be included. ETHICS AND DISSEMINATION: The study has been approved by the Swedish Ethical Review Authority (Dnr: 2020-05728). The research findings will be disseminated through peer-reviewed scientific publications and presentations at international conferences. TRIAL REGISTRATION NUMBER: NCT04995068.

Comparing supervised and unsupervised approaches to multimodal emotion recognition.

We investigated emotion classification from brief video recordings from the GEMEP database wherein actors portrayed 18 emotions. Vocal features consisted of acoustic parameters related to frequency, intensity, spectral distribution, and durations. Facial features consisted of facial action units. We first performed a series of person-independent supervised classification experiments. Best performance (AUC = 0.88) was obtained by merging the output from the best unimodal vocal (Elastic Net, AUC = 0.82) and facial (Random Forest, AUC = 0.80) classifiers using a late fusion approach and the product rule method. All 18 emotions were recognized with above-chance recall, although recognition rates varied widely across emotions (e.g., high for amusement, anger, and disgust; and low for shame). Multimodal feature patterns for each emotion are described in terms of the vocal and facial features that contributed most to classifier performance. Next, a series of exploratory unsupervised classification experiments were performed to gain more insight into how emotion expressions are organized. Solutions from traditional clustering techniques were interpreted using decision trees in order to explore which features underlie clustering. Another approach utilized various dimensionality reduction techniques paired with inspection of data visualizations. Unsupervised methods did not cluster stimuli in terms of emotion categories, but several explanatory patterns were observed. Some could be interpreted in terms of valence and arousal, but actor and gender specific aspects also contributed to clustering. Identifying explanatory patterns holds great potential as a meta-heuristic when unsupervised methods are used in complex classification tasks.

Toward Learning Machines at a Mother and Baby Unit.

Agnostic analyses of unique video material from a Mother and Baby Unit were carried out to investigate the usefulness of such analyses to the unit. The goal was to improve outcomes: the health of mothers and their babies. The method was to implement a learning machine that becomes more useful over time and over task. A feasible set-up is here described, with the purpose of producing intelligible and useful results to healthcare professionals at the unit by means of a vision processing pipeline, grouped together with multi-modal capabilities of handling annotations and audio. Algorithmic bias turned out to be an obstacle that could only partly be handled by modern pipelines for automated feature analysis. The professional use of complex quantitative scoring for various mental health-related assessments further complicated the automation of laborious tasks. Activities during the MBU stay had previously been shown to decrease psychiatric symptoms across diagnostic groups. The implementation and first set of experiments on a learning machine for the unit produced the first steps toward explaining why this is so, in turn enabling decision support to staff about what to do more and what to do less of.

Predicting treatment failure in regular care Internet-Delivered Cognitive Behavior Therapy for depression and anxiety using only weekly symptom measures.

OBJECTIVE: Therapist guided Internet-Delivered Cognitive Behavior Therapy (ICBT) is effective, but as in traditional CBT, not all patients improve, and clinicians generally fail to identify them early enough. We predict treatment failure in 12-week regular care ICBT for Depression, Panic disorder and Social anxiety disorder, using only patients' weekly symptom ratings to identify when the accuracy of predictions exceed 2 benchmarks: (a) chance, and (b) empirically derived clinician preferences for actionable predictions. METHOD: Screening, pretreatment and weekly symptom ratings from 4310 regular care ICBT-patients from the Internet Psychiatry Clinic in Stockholm, Sweden was analyzed in a series of regression models each adding 1 more week of data. Final score was predicted in a holdout test sample, which was then categorized into Success or Failure (failure defined as the absence of both remitter and responder status). Classification analyses with Balanced Accuracy and 95% Confidence intervals was then compared to predefined benchmarks. RESULTS: Benchmark 1 (better than chance) was reached 1 week into all treatments. Social anxiety disorder reached Benchmark 2 (> 65%) at week 5, whereas Depression and Panic Disorder reached it at week 6. CONCLUSIONS: For depression, social anxiety and panic disorder, prediction with only patient-rated symptom scores can detect treatment failure 6 weeks into ICBT, with enough accuracy for a clinician to take action. Early identification of failing treatment attempts may be a viable way to increase the overall success rate of existing psychological treatments by providing extra clinical resources to at-risk patients, within a so-called Adaptive Treatment Strategy. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

Supporting global health goals with information and communications technology.

The objective of this study is to critically assess the possible roles of information and communications technology (ICT) in supporting global health goals. This is done by considering privilege and connectibility. In short, ICT can contribute by providing health information via four different kinds of access, each with its own history and prospective future. All four are analyzed here, in two perspectives: business-as-usual and disruptive. Health data analytics is difficult since the digital representation of past, current, and future health information is lacking. The flow of analytics that may prove beneficial to the individual and not just meet abstract population-level goals or ambitions is analyzed in detail. Sensemaking is also needed, to meet the minimum requirement of making prospective future services understandable to policymakers. Drivers as well as barriers for areas in which policy decisions have the potential to drive positive developments for meeting the Sustainable Development Goals are identified.