Reinforcement learning for intensive care medicine: actionable clinical insights from novel approaches to reward shaping and off-policy model evaluation.

BACKGROUND: Reinforcement learning (RL) holds great promise for intensive care medicine given the abundant availability of data and frequent sequential decision-making. But despite the emergence of promising algorithms, RL driven bedside clinical decision support is still far from reality. Major challenges include trust and safety. To help address these issues, we introduce cross off-policy evaluation and policy restriction and show how detailed policy analysis may increase clinical interpretability. As an example, we apply these in the setting of RL to optimise ventilator settings in intubated covid-19 patients. METHODS: With data from the Dutch ICU Data Warehouse and using an exhaustive hyperparameter grid search, we identified an optimal set of Dueling Double-Deep Q Network RL models. The state space comprised ventilator, medication, and clinical data. The action space focused on positive end-expiratory pressure (peep) and fraction of inspired oxygen (FiO2) concentration. We used gas exchange indices as interim rewards, and mortality and state duration as final rewards. We designed a novel evaluation method called cross off-policy evaluation (OPE) to assess the efficacy of models under varying weightings between the interim and terminal reward components. In addition, we implemented policy restriction to prevent potentially hazardous model actions. We introduce delta-Q to compare physician versus policy action quality and in-depth policy inspection using visualisations. RESULTS: We created trajectories for 1118 intensive care unit (ICU) admissions and trained 69,120 models using 8 model architectures with 128 hyperparameter combinations. For each model, policy restrictions were applied. In the first evaluation step, 17,182/138,240 policies had good performance, but cross-OPE revealed suboptimal performance for 44% of those by varying the reward function used for evaluation. Clinical policy inspection facilitated assessment of action decisions for individual patients, including identification of action space regions that may benefit most from optimisation. CONCLUSION: Cross-OPE can serve as a robust evaluation framework for safe RL model implementation by identifying policies with good generalisability. Policy restriction helps prevent potentially unsafe model recommendations. Finally, the novel delta-Q metric can be used to operationalise RL models in clinical practice. Our findings offer a promising pathway towards application of RL in intensive care medicine and beyond.

Augmented intelligence facilitates concept mapping across different electronic health records.

INTRODUCTION: With the advent of artificial intelligence, the secondary use of routinely collected medical data from electronic healthcare records (EHR) has become increasingly popular. However, different EHR systems typically use different names for the same medical concepts. This obviously hampers scalable model development and subsequent clinical implementation for decision support. Therefore, converting original parameter names to a so-called ontology, a standardized set of predefined concepts, is necessary but time-consuming and labor-intensive. We therefore propose an augmented intelligence approach to facilitate ontology alignment by predicting correct concepts based on parameter names from raw electronic health record data exports. METHODS: We used the manually mapped parameter names from the multicenter "Dutch ICU data warehouse against COVID-19" sourced from three types of EHR systems to train machine learning models for concept mapping. Data from 29 intensive care units on 38,824 parameters mapped to 1,679 relevant and unique concepts and 38,069 parameters labeled as irrelevant were used for model development and validation. We used the Natural Language Toolkit (NLTK) to preprocess the parameter names based on WordNet cognitive synonyms transformed by term-frequency inverse document frequency (TF-IDF), yielding numeric features. We then trained linear classifiers using stochastic gradient descent for multi-class prediction. Finally, we fine-tuned these predictions using information on distributions of the data associated with each parameter name through similarity score and skewness comparisons. RESULTS: The initial model, trained using data from one hospital organization for each of three EHR systems, scored an overall top 1 precision of 0.744, recall of 0.771, and F1-score of 0.737 on a total of 58,804 parameters. Leave-one-hospital-out analysis returned an average top 1 recall of 0.680 for relevant parameters, which increased to 0.905 for the top 5 predictions. When reducing the training dataset to only include relevant parameters, top 1 recall was 0.811 and top 5 recall was 0.914 for relevant parameters. Performance improvement based on similarity score or skewness comparisons affected at most 5.23% of numeric parameters. CONCLUSION: Augmented intelligence is a promising method to improve concept mapping of parameter names from raw electronic health record data exports. We propose a robust method for mapping data across various domains, facilitating the integration of diverse data sources. However, recall is not perfect, and therefore manual validation of mapping remains essential.

Predicting responders to prone positioning in mechanically ventilated patients with COVID-19 using machine learning.

BACKGROUND: For mechanically ventilated critically ill COVID-19 patients, prone positioning has quickly become an important treatment strategy, however, prone positioning is labor intensive and comes with potential adverse effects. Therefore, identifying which critically ill intubated COVID-19 patients will benefit may help allocate labor resources. METHODS: From the multi-center Dutch Data Warehouse of COVID-19 ICU patients from 25 hospitals, we selected all 3619 episodes of prone positioning in 1142 invasively mechanically ventilated patients. We excluded episodes longer than 24 h. Berlin ARDS criteria were not formally documented. We used supervised machine learning algorithms Logistic Regression, Random Forest, Naive Bayes, K-Nearest Neighbors, Support Vector Machine and Extreme Gradient Boosting on readily available and clinically relevant features to predict success of prone positioning after 4 h (window of 1 to 7 h) based on various possible outcomes. These outcomes were defined as improvements of at least 10% in PaO2/FiO2 ratio, ventilatory ratio, respiratory system compliance, or mechanical power. Separate models were created for each of these outcomes. Re-supination within 4 h after pronation was labeled as failure. We also developed models using a 20 mmHg improvement cut-off for PaO2/FiO2 ratio and using a combined outcome parameter. For all models, we evaluated feature importance expressed as contribution to predictive performance based on their relative ranking. RESULTS: The median duration of prone episodes was 17 h (11-20, median and IQR, N = 2632). Despite extensive modeling using a plethora of machine learning techniques and a large number of potentially clinically relevant features, discrimination between responders and non-responders remained poor with an area under the receiver operator characteristic curve of 0.62 for PaO2/FiO2 ratio using Logistic Regression, Random Forest and XGBoost. Feature importance was inconsistent between models for different outcomes. Notably, not even being a previous responder to prone positioning, or PEEP-levels before prone positioning, provided any meaningful contribution to predicting a successful next proning episode. CONCLUSIONS: In mechanically ventilated COVID-19 patients, predicting the success of prone positioning using clinically relevant and readily available parameters from electronic health records is currently not feasible. Given the current evidence base, a liberal approach to proning in all patients with severe COVID-19 ARDS is therefore justified and in particular regardless of previous results of proning.

Right dose, right now: bedside, real-time, data-driven, and personalised antibiotic dosing in critically ill patients with sepsis or septic shock-a two-centre randomised clinical trial.

BACKGROUND: Adequate antibiotic dosing may improve outcomes in critically ill patients but is challenging due to altered and variable pharmacokinetics. To address this challenge, AutoKinetics was developed, a decision support system for bedside, real-time, data-driven and personalised antibiotic dosing. This study evaluates the feasibility, safety and efficacy of its clinical implementation. METHODS: In this two-centre randomised clinical trial, critically ill patients with sepsis or septic shock were randomised to AutoKinetics dosing or standard dosing for four antibiotics: vancomycin, ciprofloxacin, meropenem, and ceftriaxone. Adult patients with a confirmed or suspected infection and either lactate > 2 mmol/L or vasopressor requirement were eligible for inclusion. The primary outcome was pharmacokinetic target attainment in the first 24 h after randomisation. Clinical endpoints included mortality, ICU length of stay and incidence of acute kidney injury. RESULTS: After inclusion of 252 patients, the study was stopped early due to the COVID-19 pandemic. In the ciprofloxacin intervention group, the primary outcome was obtained in 69% compared to 3% in the control group (OR 62.5, CI 11.4-1173.78, p < 0.001). Furthermore, target attainment was faster (26 h, CI 18-42 h, p < 0.001) and better (65% increase, CI 49-84%, p < 0.001). For the other antibiotics, AutoKinetics dosing did not improve target attainment. Clinical endpoints were not significantly different. Importantly, higher dosing did not lead to increased mortality or renal failure. CONCLUSIONS: In critically ill patients, personalised dosing was feasible, safe and significantly improved target attainment for ciprofloxacin. TRIAL REGISTRATION: The trial was prospectively registered at Netherlands Trial Register (NTR), NL6501/NTR6689 on 25 August 2017 and at the European Clinical Trials Database (EudraCT), 2017-002478-37 on 6 November 2017.

Why we should sample sparsely and aim for a higher target: Lessons from model-based therapeutic drug monitoring of vancomycin in intensive care patients.

AIMS: To explore the optimal data sampling scheme and the pharmacokinetic (PK) target exposure on which dose computation is based in the model-based therapeutic drug monitoring (TDM) practice of vancomycin in intensive care (ICU) patients. METHODS: We simulated concentration data for 1 day following four sampling schemes, Cmin , Cmax + Cmin , Cmax + Cmid-interval + Cmin , and rich sampling where a sample was drawn every hour within a dose interval. The datasets were used for Bayesian estimation to obtain PK parameters, which were used to compute the doses for the next day based on five PK target exposures: AUC24 = 400, 500, and 600 mg·h/L and Cmin = 15 and 20 mg/L. We then simulated data for the next day, adopting the computed doses, and repeated the above procedure for 7 days. Thereafter, we calculated the percentage error and the normalized root mean square error (NRMSE) of estimated against "true" PK parameters, and the percentage of optimal treatment (POT), defined as the percentage of patients who met 400 ≤ AUC24 ≤ 600 mg·h/L and Cmin ≤ 20 mg/L. RESULTS: PK parameters were unbiasedly estimated in all investigated scenarios and the 6-day average NRMSE were 32.5%/38.5% (CL/V, where CL is clearance and V is volume of distribution) in the trough sampling scheme and 27.3%/26.5% (CL/V) in the rich sampling scheme. Regarding POT, the sampling scheme had marginal influence, while target exposure showed clear impacts that the maximum POT of 71.5% was reached when doses were computed based on AUC24 = 500 mg·h/L. CONCLUSIONS: For model-based TDM of vancomycin in ICU patients, sampling more frequently than taking only trough samples adds no value and dosing based on AUC24 = 500 mg·h/L lead to the best POT.

Optimizing Predictive Performance of Bayesian Forecasting for Vancomycin Concentration in Intensive Care Patients.

PURPOSE: Bayesian forecasting is crucial for model-based dose optimization based on therapeutic drug monitoring (TDM) data of vancomycin in intensive care (ICU) patients. We aimed to evaluate the performance of Bayesian forecasting using maximum a posteriori (MAP) estimation for model-based TDM. METHODS: We used a vancomycin TDM data set (n = 408 patients). We compared standard MAP-based Bayesian forecasting with two alternative approaches: (i) adaptive MAP which handles data over multiple iterations, and (ii) weighted MAP which weights the likelihood contribution of data. We evaluated the percentage error (PE) for seven scenarios including historical TDM data from the preceding day up to seven days. RESULTS: The mean of median PEs of all scenarios for the standard MAP, adaptive MAP and weighted MAP method were - 7.7%, -4.5% and - 6.7%. The adaptive MAP also showed the narrowest inter-quartile range of PE. In addition, regardless of MAP method, including historical TDM data further in the past will increase prediction errors. CONCLUSIONS: The proposed adaptive MAP method outperforms standard MAP in predictive performance and may be considered for improvement of model-based dose optimization. The inclusion of historical data beyond either one day (standard MAP and weighted MAP) or two days (adaptive MAP) reduces predictive performance.

Right Dose, Right Now: Development of AutoKinetics for Real Time Model Informed Precision Antibiotic Dosing Decision Support at the Bedside of Critically Ill Patients.

INTRODUCTION: Antibiotic dosing in critically ill patients is challenging because their pharmacokinetics (PK) are altered and may change rapidly with disease progression. Standard dosing frequently leads to inadequate PK exposure. Therapeutic drug monitoring (TDM) offers a potential solution but requires sampling and PK knowledge, which delays decision support. It is our philosophy that antibiotic dosing support should be directly available at the bedside through deep integration into the electronic health record (EHR) system. Therefore we developed AutoKinetics, a clinical decision support system (CDSS) for real time, model informed precision antibiotic dosing. OBJECTIVE: To provide a detailed description of the design, development, validation, testing, and implementation of AutoKinetics. METHODS: We created a development framework and used workflow analysis to facilitate integration into popular EHR systems. We used a development cycle to iteratively adjust and expand AutoKinetics functionalities. Furthermore, we performed a literature review to select and integrate pharmacokinetic models for five frequently prescribed antibiotics for sepsis. Finally, we tackled regulatory challenges, in particular those related to the Medical Device Regulation under the European regulatory framework. RESULTS: We developed a SQL-based relational database as the backend of AutoKinetics. We developed a data loader to retrieve data in real time. We designed a clinical dosing algorithm to find a dose regimen to maintain antibiotic pharmacokinetic exposure within clinically relevant safety constraints. If needed, a loading dose is calculated to minimize the time until steady state is achieved. Finally, adaptive dosing using Bayesian estimation is applied if plasma levels are available. We implemented support for five extensively used antibiotics following model development, calibration, and validation. We integrated AutoKinetics into two popular EHRs (Metavision, Epic) and developed a user interface that provides textual and visual feedback to the physician. CONCLUSION: We successfully developed a CDSS for real time model informed precision antibiotic dosing at the bedside of the critically ill. This holds great promise for improving sepsis outcome. Therefore, we recently started the Right Dose Right Now multi-center randomized control trial to validate this concept in 420 patients with severe sepsis and septic shock.

Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy.

PURPOSE: Early clinical recognition of sepsis can be challenging. With the advancement of machine learning, promising real-time models to predict sepsis have emerged. We assessed their performance by carrying out a systematic review and meta-analysis. METHODS: A systematic search was performed in PubMed, Embase.com and Scopus. Studies targeting sepsis, severe sepsis or septic shock in any hospital setting were eligible for inclusion. The index test was any supervised machine learning model for real-time prediction of these conditions. Quality of evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology, with a tailored Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) checklist to evaluate risk of bias. Models with a reported area under the curve of the receiver operating characteristic (AUROC) metric were meta-analyzed to identify strongest contributors to model performance. RESULTS: After screening, a total of 28 papers were eligible for synthesis, from which 130 models were extracted. The majority of papers were developed in the intensive care unit (ICU, n = 15; 54%), followed by hospital wards (n = 7; 25%), the emergency department (ED, n = 4; 14%) and all of these settings (n = 2; 7%). For the prediction of sepsis, diagnostic test accuracy assessed by the AUROC ranged from 0.68-0.99 in the ICU, to 0.96-0.98 in-hospital and 0.87 to 0.97 in the ED. Varying sepsis definitions limit pooling of the performance across studies. Only three papers clinically implemented models with mixed results. In the multivariate analysis, temperature, lab values, and model type contributed most to model performance. CONCLUSION: This systematic review and meta-analysis show that on retrospective data, individual machine learning models can accurately predict sepsis onset ahead of time. Although they present alternatives to traditional scoring systems, between-study heterogeneity limits the assessment of pooled results. Systematic reporting and clinical implementation studies are needed to bridge the gap between bytes and bedside.

Right Dose Right Now: bedside data-driven personalized antibiotic dosing in severe sepsis and septic shock - rationale and design of a multicenter randomized controlled superiority trial.

BACKGROUND: Antibiotic exposure is often inadequate in critically ill patients with severe sepsis or septic shock and this is associated with worse outcomes. Despite markedly altered and rapidly changing pharmacokinetics in these patients, guidelines and clinicians continue to rely on standard dosing schemes. To address this challenge, we developed AutoKinetics, a clinical decision support system for antibiotic dosing. By feeding large amounts of electronic health record patient data into pharmacokinetic models, patient-specific predicted future plasma concentrations are displayed graphically. In addition, a tailored dosing advice is provided at the bedside in real time. To evaluate the effect of AutoKinetics on pharmacometric and clinical endpoints, we are conducting the Right Dose Right Now multicenter, randomized controlled, two-arm, parallel-group, non-blinded, superiority trial. METHODS: All adult intensive care patients with a suspected or proven infection and having either lactatemia or receiving vasopressor support are eligible for inclusion. Randomization to the AutoKinetics or control group is initiated at the bedside when prescribing at least one of four commonly administered antibiotics: ceftriaxone, ciprofloxacin, meropenem and vancomycin. Dosing advice is available for patients in the AutoKinetics group, whereas patients in the control group receive standard dosing. The primary outcome of the study is pharmacometric target attainment during the first 24 h. Power analysis revealed the need for inclusion of 42 patients per group per antibiotic. Thus, a total of 336 patients will be included, 168 in each group. Secondary pharmacometric endpoints include time to target attainment and fraction of target attainment during an entire antibiotic course. Secondary clinical endpoints include mortality, clinical cure and days free from organ support. Several other exploratory and subgroup analyses are planned. DISCUSSION: This is the first randomized controlled trial to assess the effectiveness and safety of bedside data-driven automated antibiotic dosing advice. This is important as adequate antibiotic exposure may be crucial to treat severe sepsis and septic shock. In addition, the trial could prove to be a significant contribution to clinical pharmacometrics and serve as a stepping stone for the use of big data and artificial intelligence in the field. TRIAL REGISTRATION: Netherlands Trial Register (NTR), NL6501/NTR6689. Registered on 25 August 2017. European Clinical Trials Database (EudraCT), 2017-002478-37. Registered on 6 November 2017.

Clinically relevant pharmacokinetic knowledge on antibiotic dosing among intensive care professionals is insufficient: a cross-sectional study.

BACKGROUND: Antibiotic exposure in intensive care patients with sepsis is frequently inadequate and is associated with poorer outcomes. Antibiotic dosing is challenging in the intensive care, as critically ill patients have altered and fluctuating antibiotic pharmacokinetics that make current one-size-fits-all regimens unsatisfactory. Real-time bedside dosing software is not available yet, and therapeutic drug monitoring is typically used for few antibiotic classes and only allows for delayed dosing adaptation. Thus, adequate and timely antibiotic dosing continues to rely largely on the level of pharmacokinetic expertise in the ICU. Therefore, we set out to assess the level of knowledge on antibiotic pharmacokinetics among these intensive care professionals. METHODS: In May 2018, we carried out a cross-sectional study by sending out an online survey on antibiotic dosing to more than 20,000 intensive care professionals. Questions were designed to cover relevant topics in pharmacokinetics related to intensive care antibiotic dosing. The preliminary pass mark was set by members of the examination committee for the European Diploma of Intensive Care using a modified Angoff approach. The final pass mark was corrected for clinical relevance as assessed for each question by international experts on pharmacokinetics. RESULTS: A total of 1448 respondents completed the survey. Most of the respondents were intensivists (927 respondents, 64%) from 97 countries. Nearly all questions were considered clinically relevant by pharmacokinetic experts. The pass mark corrected for clinical relevance was 52.8 out of 93.7 points. Pass rates were 42.5% for intensivists, 36.1% for fellows, 24.8% for residents, and 5.8% for nurses. Scores without correction for clinical relevance were worse, indicating that respondents perform better on more relevant topics. Correct answers and concise clinical background are provided. CONCLUSIONS: Clinically relevant pharmacokinetic knowledge on antibiotic dosing among intensive care professionals is insufficient. This should be addressed given the importance of adequate antibiotic exposure in critically ill patients with sepsis. Solutions include improved education, intensified pharmacy support, therapeutic drug monitoring, or the use of real-time bedside dosing software. Questions may provide useful for teaching purposes.

External Evaluation of Population Pharmacokinetic Models of Vancomycin in Large Cohorts of Intensive Care Unit Patients.

Dosing of vancomycin is often guided by therapeutic drug monitoring and population pharmacokinetic models in the intensive care unit (ICU). The validity of these models is crucial, as ICU patients have marked pharmacokinetic variability. Therefore, we set out to evaluate the predictive performance of published population pharmacokinetic models of vancomycin in ICU patients. The PubMed database was used to search for population pharmacokinetic models of vancomycin in adult ICU patients. The identified models were evaluated in two independent data sets which were collected from two large hospitals in the Netherlands (Amsterdam UMC, Location VUmc, and OLVG Oost). We also tested a one-compartment model with fixed values for clearance and volume of distribution, in which a clinical standard dosage regimen (SDR) was mimicked to assess its predictive performance. Prediction error was calculated to assess the predictive performance of the models. Six models plus the SDR model were evaluated. The model of Roberts et al. (J. A. Roberts, F. S. Taccone, A. A. Udy, J.-L. Vincent, F. Jacobs, and J. Lipman, Antimicrob Agents Chemother 55:2704-2709, 2011, https://doi.org/10.1128/AAC.01708-10) performed satisfactorily, with mean and median values of prediction error of 5.1% and -7.5%, respectively, for Amsterdam UMC, Location VUmc, patients, and -12.6% and -17.2% respectively, for OLVG Oost patients. The other models, including the SDR model, yielded high mean values (-49.7% to 87.7%) and median values (-56.1% to 66.1%) for both populations. In conclusion, only the model of Roberts et al. was able to validly predict the concentrations of vancomycin for our data, whereas other models and standard dosing were largely inadequate. Extensive evaluation should precede the adoption of any model in clinical practice for ICU patients.