Can Machine Learning Personalize Cardiovascular Therapy in Sepsis?

Large randomized trials in sepsis have generally failed to find effective novel treatments. This is increasingly attributed to patient heterogeneity, including heterogeneous cardiovascular changes in septic shock. We discuss the potential for machine learning systems to personalize cardiovascular resuscitation in sepsis. While the literature is replete with proofs of concept, the technological readiness of current systems is low, with a paucity of clinical trials and proven patient benefit. Systems may be vulnerable to confounding and poor generalization to new patient populations or contemporary patterns of care. Typical electronic health records do not capture rich enough data, at sufficient temporal resolution, to produce systems that make actionable treatment suggestions. To resolve these issues, we recommend a simultaneous focus on technical challenges and removing barriers to translation. This will involve improving data quality, adopting causally grounded models, prioritizing safety assessment and integration into healthcare workflows, conducting randomized clinical trials and aligning with regulatory requirements.

Quantifying the impact of AI recommendations with explanations on prescription decision making.

The influence of AI recommendations on physician behaviour remains poorly characterised. We assess how clinicians' decisions may be influenced by additional information more broadly, and how this influence can be modified by either the source of the information (human peers or AI) and the presence or absence of an AI explanation (XAI, here using simple feature importance). We used a modified between-subjects design where intensive care doctors (N = 86) were presented on a computer for each of 16 trials with a patient case and prompted to prescribe continuous values for two drugs. We used a multi-factorial experimental design with four arms, where each clinician experienced all four arms on different subsets of our 24 patients. The four arms were (i) baseline (control), (ii) peer human clinician scenario showing what doses had been prescribed by other doctors, (iii) AI suggestion and (iv) XAI suggestion. We found that additional information (peer, AI or XAI) had a strong influence on prescriptions (significantly for AI, not so for peers) but simple XAI did not have higher influence than AI alone. There was no correlation between attitudes to AI or clinical experience on the AI-supported decisions and nor was there correlation between what doctors self-reported about how useful they found the XAI and whether the XAI actually influenced their prescriptions. Our findings suggest that the marginal impact of simple XAI was low in this setting and we also cast doubt on the utility of self-reports as a valid metric for assessing XAI in clinical experts.

Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI.

A growing number of artificial intelligence (AI)-based clinical decision support systems are showing promising performance in preclinical, in silico evaluation, but few have yet demonstrated real benefit to patient care. Early-stage clinical evaluation is important to assess an AI system's actual clinical performance at small scale, ensure its safety, evaluate the human factors surrounding its use and pave the way to further large-scale trials. However, the reporting of these early studies remains inadequate. The present statement provides a multi-stakeholder, consensus-based reporting guideline for the Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence (DECIDE-AI). We conducted a two-round, modified Delphi process to collect and analyze expert opinion on the reporting of early clinical evaluation of AI systems. Experts were recruited from 20 pre-defined stakeholder categories. The final composition and wording of the guideline was determined at a virtual consensus meeting. The checklist and the Explanation & Elaboration (E&E) sections were refined based on feedback from a qualitative evaluation process. In total, 123 experts participated in the first round of Delphi, 138 in the second round, 16 in the consensus meeting and 16 in the qualitative evaluation. The DECIDE-AI reporting guideline comprises 17 AI-specific reporting items (made of 28 subitems) and ten generic reporting items, with an E&E paragraph provided for each. Through consultation and consensus with a range of stakeholders, we developed a guideline comprising key items that should be reported in early-stage clinical studies of AI-based decision support systems in healthcare. By providing an actionable checklist of minimal reporting items, the DECIDE-AI guideline will facilitate the appraisal of these studies and replicability of their findings.

The diagnostic certainty levels of junior clinicians: A retrospective cohort study.

BACKGROUND: Clinical decision-making is influenced by many factors, including clinicians' perceptions of the certainty around what is the best course of action to pursue. OBJECTIVE: To characterise the documentation of working diagnoses and the associated level of real-time certainty expressed by clinicians and to gauge patient opinion about the importance of research into clinician decision certainty. METHOD: This was a single-centre retrospective cohort study of non-consultant grade clinicians and their assessments of patients admitted from the emergency department between 01 March 2019 and 31 March 2019. De-identified electronic health record proformas were extracted that included the type of diagnosis documented and the certainty adjective used. Patient opinion was canvassed from a focus group. RESULTS: During the study period, 850 clerking proformas were analysed; 420 presented a single diagnosis, while 430 presented multiple diagnoses. Of the 420 single diagnoses, 67 (16%) were documented as either a symptom or physical sign and 16 (4%) were laboratory-result-defined diagnoses. No uncertainty was expressed in 309 (74%) of the diagnoses. Of 430 multiple diagnoses, uncertainty was expressed in 346 (80%) compared to 84 (20%) in which no uncertainty was expressed. The patient focus group were unanimous in their support of this research. CONCLUSION: The documentation of working diagnoses is highly variable among non-consultant grade clinicians. In nearly three quarters of assessments with single diagnoses, no element of uncertainty was implied or quantified. More uncertainty was expressed in multiple diagnoses than single diagnoses. IMPLICATIONS: Increased standardisation of documentation will help future studies to better analyse and quantify diagnostic certainty in both single and multiple working diagnoses. This could lead to subsequent examination of their association with important process or clinical outcome measures.

Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies.

OBJECTIVE: To systematically examine the design, reporting standards, risk of bias, and claims of studies comparing the performance of diagnostic deep learning algorithms for medical imaging with that of expert clinicians. DESIGN: Systematic review. DATA SOURCES: Medline, Embase, Cochrane Central Register of Controlled Trials, and the World Health Organization trial registry from 2010 to June 2019. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: Randomised trial registrations and non-randomised studies comparing the performance of a deep learning algorithm in medical imaging with a contemporary group of one or more expert clinicians. Medical imaging has seen a growing interest in deep learning research. The main distinguishing feature of convolutional neural networks (CNNs) in deep learning is that when CNNs are fed with raw data, they develop their own representations needed for pattern recognition. The algorithm learns for itself the features of an image that are important for classification rather than being told by humans which features to use. The selected studies aimed to use medical imaging for predicting absolute risk of existing disease or classification into diagnostic groups (eg, disease or non-disease). For example, raw chest radiographs tagged with a label such as pneumothorax or no pneumothorax and the CNN learning which pixel patterns suggest pneumothorax. REVIEW METHODS: Adherence to reporting standards was assessed by using CONSORT (consolidated standards of reporting trials) for randomised studies and TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) for non-randomised studies. Risk of bias was assessed by using the Cochrane risk of bias tool for randomised studies and PROBAST (prediction model risk of bias assessment tool) for non-randomised studies. RESULTS: Only 10 records were found for deep learning randomised clinical trials, two of which have been published (with low risk of bias, except for lack of blinding, and high adherence to reporting standards) and eight are ongoing. Of 81 non-randomised clinical trials identified, only nine were prospective and just six were tested in a real world clinical setting. The median number of experts in the comparator group was only four (interquartile range 2-9). Full access to all datasets and code was severely limited (unavailable in 95% and 93% of studies, respectively). The overall risk of bias was high in 58 of 81 studies and adherence to reporting standards was suboptimal (<50% adherence for 12 of 29 TRIPOD items). 61 of 81 studies stated in their abstract that performance of artificial intelligence was at least comparable to (or better than) that of clinicians. Only 31 of 81 studies (38%) stated that further prospective studies or trials were required. CONCLUSIONS: Few prospective deep learning studies and randomised trials exist in medical imaging. Most non-randomised trials are not prospective, are at high risk of bias, and deviate from existing reporting standards. Data and code availability are lacking in most studies, and human comparator groups are often small. Future studies should diminish risk of bias, enhance real world clinical relevance, improve reporting and transparency, and appropriately temper conclusions. STUDY REGISTRATION: PROSPERO CRD42019123605.

Gaps in patient-reported outcome measures in randomized clinical trials of cardiac catheter ablation: a systematic review.

The aim of this systematic review was to evaluate randomized clinical trials (RCTs) of cardiac catheter ablation (CCA) and to assess the prevalence, characteristics and reporting standards of clinically relevant patient-reported outcome measures (PROMs). Electronic database searches of Medline, Embase, CENTRAL, and the WHO Trial Registry were conducted in March 2019. The study protocol was registered on PROSPERO (CRD42019133086). Of 7125 records identified, 237 RCTs were included for analysis, representing 35 427 patients with a mean age of 59 years. Only 43 RCTs (18%) reported PROMs of which 27 included a generic PROM that measured health-related quality of life (HRQL) necessary to conduct comparative effectiveness research. There was notable under-representation of certain patient groups-only 31% were women and only 8% were of non-Caucasian ethnicity, in trials which reported such data. The reporting standard of PROMs was highly variable with 8-62% adherence against CONSORT PRO-specific items. In summary, PROMs play a crucial role in determining the clinical and cost-effectiveness of treatments which primarily offer symptomatic improvement, such as CCA. Their underuse significantly limits evaluation of the comparative effectiveness of treatments. Using CCA as an exemplar, there are additional issues of infrequent assessment, poor reporting and under-representation of many population groups. Greater use of PROMs, and specifically validated HRQL questionnaires, is paramount in giving patients a voice in studies, generating more meaningful comparisons between treatments and driving better patient-centred clinical and policy-level decision-making.

Real time self-rating of decision certainty by clinicians: a systematic review.

BackgroundWe sought to establish to what extent decision certainty has been measured in real time and whether high or low levels of certainty correlate with clinical outcomes.MethodsOur pre-specified study protocol is published on PROSPERO, CRD42019128112. We identified prospective studies from Medline, Embase and PsycINFO up to February 2019 that measured real time self-rating of the certainty of a medical decision by a clinician.FindingsNine studies were included and all were generally at high risk of bias. Only one study assessed long-term clinical outcomes: patients rated with high diagnostic uncertainty for heart failure had longer length of stay, increased mortality and higher readmission rates at 1 year than those rated with diagnostic certainty. One other study demonstrated the danger of extreme diagnostic confidence - 7% of cases (24/341) labelled as having either 0% or 100% diagnostic likelihood of heart failure were made in error.ConclusionsThe literature on real time self-rated certainty of clinician decisions is sparse and only relates to diagnostic decisions. Further prospective research with a view to generating hypotheses for testable interventions that can better calibrate clinician certainty with accuracy of decision making could be valuable in reducing diagnostic error and improving outcomes.

Vasopressin in septic shock: an individual patient data meta-analysis of randomised controlled trials.

PURPOSE: We performed an individual patient data meta-analysis to investigate the possible benefits and harms of vasopressin therapy in adults with septic shock both overall and in pre-defined subgroups. METHODS: Our pre-specified study protocol is published on PROSPERO, CRD42017071698. We identified randomised clinical trials up to January 2019 investigating vasopressin therapy versus any other vasoactive comparator in adults with septic shock. Individual patient data from each trial were compiled. Conventional two-stage meta-analyses were performed as well as one-stage regression models with single treatment covariate interactions for subgroup analyses. RESULTS: Four trials were included with a total of 1453 patients. For the primary outcomes, there was no effect of vasopressin on 28-day mortality [relative risk (RR) 0.98, 95% CI 0.86-1.12] or serious adverse events (RR 1.02, 95% CI 0.82-1.26). Vasopressin led to more digital ischaemia [absolute risk difference (ARD) 1.7%, 95% CI 0.3%-3.2%] but fewer arrhythmias (ARD - 2.8%, 95% CI - 0.2% to - 5.3%). Mesenteric ischaemia and acute coronary syndrome events were similar between groups. Vasopressin reduced the requirement for renal replacement therapy (RRT) (RR 0.86, 95% CI 0.74-0.99), but this finding was not robust to sensitivity analyses. There were no statistically significant interactions in the pre-defined subgroups (baseline kidney injury severity, baseline lactate, baseline norepinephrine requirement and time to study inclusion). CONCLUSIONS: Vasopressin therapy in septic shock had no effect on 28-day mortality although the confidence intervals are wide. It appears safe but with a different side effect profile from norepinephrine. The finding on reduced RRT should be interpreted cautiously. Future trials should focus on long-term outcomes in select patient groups as well as incorporating cost effectiveness analyses regarding possible reduced RRT use.

Rapid Induction of Liver Regeneration for Major Hepatectomy (REBIRTH): A Randomized Controlled Trial of Portal Vein Embolisation versus ALPPS Assisted with Radiofrequency.

To avoid liver insufficiency following major hepatic resection, portal vein embolisation (PVE) is used to induce liver hypertrophy pre-operatively. Associating liver partition with portal vein ligation for staged hepatectomy assisted with radiofrequency (RALPPS) was introduced as an alternative method. A randomized controlled trial comparing PVE with RALPPS for the pre-operative manipulation of liver volume in patients with a future liver remnant volume (FLRV) ≤25% (or ≤35% if receiving preoperative chemotherapy) was conducted. The primary endpoint was increase in size of the FLRV. The secondary endpoints were length of time taken for the volume gain, morbidity, operation length and post-operative liver function. Between July 2015 and October 2017, 57 patients were randomised to RALPPS (n = 29) and PVE (n = 28). The mean percentage of increase in the FLRV was 80.7 ± 13.7% after a median 20 days following RALPPS compared to 18.4 ± 9.8% after 35 days (p < 0.001) following PVE. Twenty-four patients after RALPPS and 21 after PVE underwent stage-2 operation. Final resection was achieved in 92.3% and 66.6% patients in RALPPS and PVE, respectively (p = 0.007). There was no difference in morbidity, and one 30-day mortality after RALPPS (p = 0.991) was reported. RALPPS is more effective than PVE in increasing FLRV and the number of patients for surgical resection.

Financial performance of English NHS trusts and variation in clinical outcomes: a longitudinal observational study.

OBJECTIVES: To examine the association between financial performance as measured by operating margin (surplus/deficit as a proportion of turnover) and clinical outcomes in English National Health Service (NHS) trusts. SETTING: Longitudinal, observational study in 149 acute NHS trusts in England between the financial years 2011 and 2016. PARTICIPANTS: Our analysis focused on outcomes at individual NHS Trust-level (composed of one or more acute hospitals). PRIMARY AND SECONDARY OUTCOMES: Outcome measures included readmissions, inpatient satisfaction score and the following process measures: emergency department (Accident and Emergency (A&E)) waiting time targets, cancer referral and treatment targets and delayed transfers of care (DTOCs). RESULTS: There was a progressive increase in the proportion of trusts in financial deficit: 22% in 2011, 27% in 2012, 28% in 2013, 51% in 2014, 68% in 2015 and 91% in 2016. In linear regression analyses, there was no significant association between operating margin and clinical outcomes (readmission rate or inpatient satisfaction score). There was, however, a significant association between operating margin and process measures (DTOCs, A&E breaches and cancer waiting time targets). Between the best and worst financially performing Trusts, there was an approximately 2-fold increase in A&E breaches and DTOCs overall although this variation decreased over the 6 years. Between the best and worst performing trusts on cancer targets, the magnitude of difference was smaller (1.16 and 1.15-fold), although the variation slowly rose during the 6 years. CONCLUSIONS: Operating margins in English NHS trusts progressively worsened during 2011-2016, and this change was associated with poorer performance on several process measures but not with hospital readmissions or inpatient satisfaction. Significant variation exists between the best and worst financially performing Trusts. Further research is needed to examine the causal nature of relationships between financial performance, process measures and outcomes.

Statin therapy for acute respiratory distress syndrome: an individual patient data meta-analysis of randomised clinical trials.

PURPOSE: We performed an individual patient data meta-analysis to assess the possible benefits and harms of statin therapy in adults with acute respiratory distress syndrome (ARDS) and to investigate effects in specific ARDS subgroups. METHODS: We identified randomised clinical trials up to 31 October 2016 that had investigated statin therapy versus placebo in patients with ARDS. Individual patient data from each trial were compiled. Conventional two-stage meta-analyses were performed for primary and secondary outcomes, and one-stage regression models with single treatment-covariate interactions for subgroup analyses. Risk of bias was assessed using the Cochrane Risk of Bias Tool. RESULTS: Six trials with a total of 1755 patients were included. For the primary outcomes, there was no significant effect of statin therapy on 28-day mortality [relative risk (RR) 1.03, 95% CI 0.86-1.23], ventilator-free days (mean difference 0.34 days, 95% CI -0.68 to 1.36) or serious adverse events (RR 1.14, 95% CI 0.84-1.53). There was a significantly increased incidence of raised serum creatine kinase or transaminase levels with statin therapy (106/879; 12.1%) versus control (78/876; 8.9%) (RR 1.40, 95% CI 1.07-1.83, p = 0.015). There were no significant treatment-covariate interactions in the predefined subgroups investigated. CONCLUSIONS: We found no clinical benefit from initiation of statin therapy in adult patients with ARDS, either overall or in predefined subgroups. While there was an increased incidence of raised serum creatine kinase and transaminase levels, there was no difference in serious adverse events among groups. Therefore, we do not recommend initiation of statin therapy for the treatment of ARDS.

Very large treatment effects in randomised trials as an empirical marker to indicate whether subsequent trials are necessary: meta-epidemiological assessment.

OBJECTIVE:  To examine whether a very large effect (VLE; defined as a relative risk of ≤0.2 or ≥5) in a randomised trial could be an empirical marker that subsequent trials are unnecessary. DESIGN:  Meta-epidemiological assessment of existing published data on randomised trials. DATA SOURCES:  Cochrane Database of Systematic Reviews (2010, issue 7) with data on subsequent large trials updated to 2015, issue 12. ELIGIBILITY CRITERIA:  All binary outcome forest plots were selected, which contained an index randomised trial with a VLE that was nominally statistically significant (P<0.05), included a subsequent large randomised trial (≥200 events and ≥200 non-events) for validation of the effect, assessed a primary outcome of the review, and was not a subgroup or sensitivity analysis. RESULTS:  Of 3082 reviews yielding 85 002 forest plots, only 44 (0.05%) satisfied the inclusion criteria. Index trials were generally small, with a median sample of 99 (median 14 events). Few index trials were rated at low risk of bias (9 of 44; 20%). The relative risk was closer to the null in the subsequent large trials in 43 of 44 cases. Subsequent large trial data failed to find a statistically significant (P<0.05) effect in the same direction in 19 cases (43%, 95% confidence interval 29% to 58%). Even when the subsequent large trials did find a significant effect in the same direction, the additional primary outcomes in most of these trials would have to be considered before deciding in favour of using the intervention. Subsequent large trial data found a statistically significant effect in the same direction in 19 of 21 cases when the index trial also had a value of P<0.001. CONCLUSIONS:  The frequency of VLEs followed by a large trial is vanishingly small, and where they occur they do not appear to be a reliable marker for a benefit that is reproducible and directly actionable. An empirical rule using a VLE in a randomised controlled trial as a marker that further trials are unnecessary would be neither practical nor useful. Caution should be taken when interpreting small studies with very large treatment effects.

Mortality Among Older Adults Before Versus After Hospital Transition to Intensivist Staffing.

BACKGROUND: A large body of research suggests that hospitals with intensive care units staffed by board-certified intensivists have lower mortality rates than those that do not. OBJECTIVE: To determine whether hospitals can reduce their mortality by adopting an intensivist staffing model. DESIGN: Retrospective, longitudinal study using 2003-2010 Medicare data and the Leapfrog Group Hospital surveys. SETTING AND PATIENTS: In total, 2,916,801 Medicare patients at 488 US hospitals. MEASUREMENTS: We studied 30-day and in-hospital mortality among patients with several common medical and surgical conditions. We first compared risk-adjusted mortality rates of 3 groups of hospitals: those that were intensivist staffed throughout this time period, those that were not intensivist staffed, and those that transitioned to intensivist staffing somewhere during the period. We then examined rates of mortality improvement within each of the 3 groups and used difference-in-differences techniques to assess the independent effect of intensivist staffing among the subset of hospitals that transitioned. RESULTS: Hospitals with intensivist staffing at the beginning of our study period had lower mortality rates than those without. However, hospitals that adopted intensivist staffing during the study period did not substantially improve their mortality rates. In our difference-in-differences analysis, there was no significant independent improvement in mortality after transitioning to intensivist staffing either overall [relative risk (RR), 0.96; 95% confidence interval (CI), 0.90-1.02] or in the medical (RR, 0.95; 95% CI, 0.89-1.02) or surgical populations (RR, 0.97; 95% CI, 0.84-1.10). LIMITATIONS: Risk adjustment was based on administrative data. Categorization of exposure was by survey response at the hospital level. CONCLUSIONS: Adoption of an intensivist staffing model was not associated with improved mortality in Medicare beneficiaries. These findings suggest that the lower mortality rates previously observed at hospitals with intensivist staffing may be attributable to other factors.

Comparative safety and efficacy of vasopressors for mortality in septic shock: A network meta-analysis.

INTRODUCTION: Septic shock is a life-threatening condition requiring vasopressor agents to support the circulatory system. Several agents exist with choice typically guided by the specific clinical scenario. We used a network meta-analysis approach to rate the comparative efficacy and safety of vasopressors for mortality and arrhythmia incidence in septic shock patients. METHODS: We performed a comprehensive electronic database search including Medline, Embase, Science Citation Index Expanded and the Cochrane database. Randomised trials investigating vasopressor agents in septic shock patients and specifically assessing 28-day mortality or arrhythmia incidence were included. A Bayesian network meta-analysis was performed using Markov chain Monte Carlo methods. RESULTS: Thirteen trials of low to moderate risk of bias in which 3146 patients were randomised were included. There was no pairwise evidence to suggest one agent was superior over another for mortality. In the network meta-analysis, vasopressin was significantly superior to dopamine (OR 0.68 (95% CI 0.5 to 0.94)) for mortality. For arrhythmia incidence, standard pairwise meta-analyses confirmed that dopamine led to a higher incidence of arrhythmias than norepinephrine (OR 2.69 (95% CI 2.08 to 3.47)). In the network meta-analysis, there was no evidence of superiority of one agent over another. CONCLUSIONS: In this network meta-analysis, vasopressin was superior to dopamine for 28-day mortality in septic shock. Existing pairwise information supports the use of norepinephrine over dopamine. Our findings suggest that dopamine should be avoided in patients with septic shock and that other vasopressor agents should continue to be based on existing guidelines and clinical judgement of the specific presentation of the patient.

Factors associated with early provision of respiratory support in deteriorating ward patients with chest sepsis.

BACKGROUND: Implementation of the National Early Warning Score in the National Health Service (NHS) has renewed focus on prompt identification and referral of the deteriorating ward patient. A large body of published work suggests that delay in both referral and admission to critical care can be associated with poor outcomes. We sought to explore factors associated with early provision of respiratory support in a cohort of deteriorating ward patients referred to the critical care team. METHODS: SPOT(light) (Sepsis Pathophysiological and Organisational Timing) was a prospective observational study carried out between 2010 and 2011 in acute NHS hospitals in the UK. In a pilot retrospective analysis, we merged data from this study with organ support data from the Critical Care Minimum Data Set. Deteriorating adult ward patients referred to the critical care team with presumed severe chest sepsis and with no treatment limitations in place were eligible for inclusion. We used these data to assess critical care bed availabilty and factors affecting decisions to accept patients to the intensive care unit. FINDINGS: 828 patients at 13 acute hospitals were referred to the critical care team. 7-day mortality was 17% (138 patients); 115 (83%) of these patients had not received inspiratory positive pressure ventilation (IPPV) despite having had no treatment limitations in place. 275 (33%) of the 828 patients were accepted by the critical team after review. A decision to accept was significantly more likely when beds were available than when not available (269/275 [34%] vs 6/275 [15%], p=0·010). Mean time to commencing IPPV was significantly reduced by critical care bed availability (0·8 days [SD 1·3] vs 2·3 [1·4], p=0·001). 130 patients (16%) received IPPV, of whom 93 (72%) proceeded directly to IPPV rather than via non-invasive ventilation (NIV) initially. Patients were more likely to proceed directly to IPPV where the critical care team made a decision to admit (72/93 [77%] vs 21/93 [57%], p=0·018). INTERPRETATION: These pilot data suggest that critical care bed availability and a decision to admit to critical care are associated with both a faster and a more direct provision of IPPV (rather than via NIV initially). In this small sample, a large proportion of the mortality occurred in patients who had not received IPPV despite having had no treatment limitations in place. FUNDING: Wellcome Trust, National Institute for Health Service Support Costs, Intensive Care National Audit & Research Centre.