Local Modeling by Adapting Source Calibration Models to Analyte Shifted Target Domain Samples Without Reference Values.

Spectral multivariate calibration aims to derive models characterizing mathematical relationships between sample analyte amounts and corresponding spectral responses. These models are effective at predicting target domain sample analyte amounts when target samples are within the analyte and spectral calibration source domain. Models fail when target samples shift (analyte amounts and/or spectra) from the original calibration domain model. A total recalibration solution requires acquisition of new sample reference values and spectra. However, obtaining enough reference values to distinguish the target domain may be challenging or expensive. A simpler approach adapts the original model to the target domain using target sample spectra without analyte reference values (unlabeled). Analytical chemists have developed several machine learning algorithms using unlabeled regression domain adaptation processes. Unfortunately, prediction accuracy declines for these methods depending on how much the target domain analyte distribution has shifted from the calibration distribution, and regression transfer learning methods are instead needed. Regression domain adaptation and transfer learning are often referred to as model updating in analytical chemistry, but regression domain adaptation only applies to spectral shifts. The regression transfer learning method presented in this paper named null augmentation regression constant analyte (NARCA) leverages unlabeled repeat spectra of a single target sample to update an original calibration model to the shifted target domain sample. With sample repeat spectra, the analyte amount can be assumed constant or nearly constant for NARCA and because models are formed for one sample, NARCA operates as a local modeling method. The performance of NARCA as a regression transfer learning method is evaluated using five near-infrared data sets.

Improved Finite Element Model Updating of a Highway Viaduct Using Acceleration and Strain Data.

Most finite element model updating (FEMU) studies on bridges are acceleration-based due to their lower cost and ease of use compared to strain- or displacement-based methods, which entail costly experiments and traffic disruptions. This leads to a scarcity of comprehensive studies incorporating strain measurements. This study employed the strain- and acceleration-based FEMU analyses performed on a more than 50-year-old multi-span concrete highway viaduct. Mid-span strains under heavy vehicles were considered for the strain-based FEMU, and frequencies and mode shapes for the acceleration-based FEMU. The analyses were performed separately for up to three variables, representing Young's modulus adjustment factors for different groups of structural elements. FEMU studies considered residual minimisation and the error-domain model falsification (EDMF) methodology. The residual minimisation utilised four different single-objective optimisations focusing on strains, frequencies, and mode shapes. Strain- and frequency-based FEMU analyses resulted in an approximately 20% increase in the overall superstructure's design stiffness. This study shows the benefits of the intuitive EDMF over residual minimisation for FEMU, where information gained from the strain data, in addition to the acceleration data, manifests more sensible updated variables. EDMF finally resulted in a 25-50% overestimated design stiffness of internal main girders.

Don't be misled: 3 misconceptions about external validation of clinical prediction models.

Clinical prediction models provide risks of health outcomes that can inform patients and support medical decisions. However, most models never make it to actual implementation in practice. A commonly heard reason for this lack of implementation is that prediction models are often not externally validated. While we generally encourage external validation, we argue that an external validation is often neither sufficient nor required as an essential step before implementation. As such, any available external validation should not be perceived as a license for model implementation. We clarify this argument by discussing 3 common misconceptions about external validation. We argue that there is not one type of recommended validation design, not always a necessity for external validation, and sometimes a need for multiple external validations. The insights from this paper can help readers to consider, design, interpret, and appreciate external validation studies.

Sustainable deployment of clinical prediction tools-a 360° approach to model maintenance.

BACKGROUND: As the enthusiasm for integrating artificial intelligence (AI) into clinical care grows, so has our understanding of the challenges associated with deploying impactful and sustainable clinical AI models. Complex dataset shifts resulting from evolving clinical environments strain the longevity of AI models as predictive accuracy and associated utility deteriorate over time. OBJECTIVE: Responsible practice thus necessitates the lifecycle of AI models be extended to include ongoing monitoring and maintenance strategies within health system algorithmovigilance programs. We describe a framework encompassing a 360° continuum of preventive, preemptive, responsive, and reactive approaches to address model monitoring and maintenance from critically different angles. DISCUSSION: We describe the complementary advantages and limitations of these four approaches and highlight the importance of such a coordinated strategy to help ensure the promise of clinical AI is not short-lived.

Comparing Predictive Performance of Time Invariant and Time Variant Clinical Prediction Models in Cardiac Surgery.

Clinical prediction models are increasingly used across healthcare to support clinical decision making. Existing methods and models are time-invariant and thus ignore the changes in populations and healthcare practice that occur over time. We aimed to compare the performance of time-invariant with time-variant models in UK National Adult Cardiac Surgery Audit data from Manchester University NHS Foundation Trust between 2009 and 2019. Data from 2009-2011 were used for initial model fitting, and data from 2012-2019 for validation and updating. We fitted four models to the data: a time-invariant logistic regression model (not updated), a logistic model which was updated every year and validated it in each subsequent year, a logistic regression model where the intercept is a function of calendar time (not updated), and a continually updating Bayesian logistic model which was updated with each new observation and continuously validated. We report predictive performance over the complete validation cohort and for each year in the validation data. Over the complete validation data, the Bayesian model had the best predictive performance.

A target tracking method based on adaptive occlusion judgment and model updating strategy.

Target tracking is an important research in the field of computer vision. Despite the rapid development of technology, difficulties still remain in balancing the overall performance for target occlusion, motion blur, etc. To address the above issue, we propose an improved kernel correlation filter tracking algorithm with adaptive occlusion judgement and model updating strategy (called Aojmus) to achieve robust target tracking. Firstly, the algorithm fuses color-naming (CN) and histogram of gradients (HOG) features as a feature extraction scheme and introduces a scale filter to estimate the target scale, which reduces tracking error caused by the variations of target features and scales. Secondly, the Aojmus introduces four evaluation indicators and a double thresholding mechanism to determine whether the target is occluded and the degree of occlusion respectively. The four evaluation results are weighted and fused to a final value. Finally, the updating strategy of the model is adaptively adjusted based on the weighted fusion value and the result of the scale estimation. Experimental evaluations on the OTB-2015 dataset are conducted to compare the performance of the Aojmus algorithm with four other comparable algorithms in terms of tracking precision, success rate, and speed. The experimental results show that the proposed Aojmus algorithm outperforms all the algorithms compared in terms of tracking precision. The Aojmus also exhibits excellent performance on attributes such as target occlusion and motion blur in terms of success rate. In addition, the processing speed reaches 74.85 fps, which also demonstrates good real-time performance.

Dynamic updating of clinical survival prediction models in a changing environment.

BACKGROUND: Over time, the performance of clinical prediction models may deteriorate due to changes in clinical management, data quality, disease risk and/or patient mix. Such prediction models must be updated in order to remain useful. In this study, we investigate dynamic model updating of clinical survival prediction models. In contrast to discrete or one-time updating, dynamic updating refers to a repeated process for updating a prediction model with new data. We aim to extend previous research which focused largely on binary outcome prediction models by concentrating on time-to-event outcomes. We were motivated by the rapidly changing environment seen during the COVID-19 pandemic where mortality rates changed over time and new treatments and vaccines were introduced. METHODS: We illustrate three methods for dynamic model updating: Bayesian dynamic updating, recalibration, and full refitting. We use a simulation study to compare performance in a range of scenarios including changing mortality rates, predictors with low prevalence and the introduction of a new treatment. Next, the updating strategies were applied to a model for predicting 70-day COVID-19-related mortality using patient data from QResearch, an electronic health records database from general practices in the UK. RESULTS: In simulated scenarios with mortality rates changing over time, all updating methods resulted in better calibration than not updating. Moreover, dynamic updating outperformed ad hoc updating. In the simulation scenario with a new predictor and a small updating dataset, Bayesian updating improved the C-index over not updating and refitting. In the motivating example with a rare outcome, no single updating method offered the best performance. CONCLUSIONS: We found that a dynamic updating process outperformed one-time discrete updating in the simulations. Bayesian updating offered good performance overall, even in scenarios with new predictors and few events. Intercept recalibration was effective in scenarios with smaller sample size and changing baseline hazard. Refitting performance depended on sample size and produced abrupt changes in hazard ratio estimates between periods.

An Uncertainty Model for Strain Gages Using Monte Carlo Methodology.

For the purpose of validation and identification of mechanical systems, measurements are indispensable. However, they require knowledge of the inherent uncertainty to provide valid information. This paper describes a method on how to evaluate uncertainties in strain measurement using electric strain gages for practical engineering applications. Therefore, a basic model of the measurement is deduced that comprises the main influence factors and their uncertainties. This is performed using the example of a project dealing with strain measurement on the concrete surface of a large-span road bridge under static loading. Special attention is given to the statistical modeling of the inputs, the underlying physical relationship, and the incorporation and the impact of nonlinearities for different environmental conditions and strain levels. In this regard, also experiments were conducted to quantify the influence of misalignment of the gages. The methodological approach used is Monte Carlo simulation. A subsequent variance-based sensitivity analysis reveals the degree of nonlinearity in the relationship and the importance of the different factors to the resulting probability distribution. The developed scheme requires a minimum of expert knowledge of the analytical derivation of measurement uncertainties and can easily be modified for differing requirements and purposes.

Bridge Model Updating Based on Wavelet Neural Network and Wind-Driven Optimization.

Aging, corrosive environments, and inadequate maintenance may result in performance deterioration of civil infrastructures, and finite element model updating is a commonly employed structural health monitoring procedure in civil engineering to reflect the current situation and to ensure the safety and serviceability of structures. Using the finite element model updating process to obtain the relationship between the structural responses and updating parameters, this paper proposes a method of using the wavelet neural network (WNN) as the surrogate model combined with the wind-driven optimization (WDO) algorithm to update the structural finite element model. The method was applied to finite element model updating of a continuous beam structure of three equal spans to verify its feasibility, the results show that the WNN can reflect the nonlinear relationship between structural responses and the parameters and has an outstanding simulation performance; the WDO has an excellent ability for optimization and can effectively improve the efficiency of model updating. Finally, the method was applied to update a real bridge model, and the results show that the finite element model update based on WDO and WNN is applicable to the updating of a multi-parameter bridge model, which has practical significance in engineering and high efficiency in finite element model updating. The differences between the updated values and measured values are all within the range of 5%, while the maximum difference was reduced from -10.9% to -3.6%. The proposed finite element model updating method is applicable and practical for multi-parameter bridge model updating and has the advantages of high updating efficiency, reliability, and practical significance.

Dynamic prediction modeling of postoperative mortality among patients undergoing surgical aortic valve replacement in a statewide cohort over a 12-year period.

Objective: Clinical prediction models for surgical aortic valve replacement mortality, are valuable decision tools but are often limited in their ability to account for changes in medical practice, patient selection, and the risk of outcomes over time. Recent research has identified methods to update models as new data accrue, but their effect on model performance has not been rigorously tested. Methods: The study population included 44,546 adults who underwent an isolated surgical aortic valve replacement from January 1, 1999, to December 31, 2018, statewide in Pennsylvania. After chronologically splitting the data into training and validation sets, we compared calibration, discrimination, and accuracy measures amongst a nonupdating model to 2 methods of model updating: calibration regression and the novel dynamic logistic state space model. Results: The risk of mortality decreased significantly during the validation period (P < .01) and the nonupdating model demonstrated poor calibration and reduced accuracy over time. Both updating models maintained better calibration (Hosmer-Lemeshow χ2 statistic) than the nonupdating model: nonupdating (156.5), calibration regression (4.9), and dynamic logistic state space model (8.0). Overall accuracy (Brier score) was consistently better across both updating models: dynamic logistic state space model (0.0252), calibration regression (0.0253), and nonupdating (0.0256). Discrimination improved with the dynamic logistic state space model (area under the curve, 0.696) compared with the nonupdating model (area under the curve, 0.685) and calibration regression method (area under the curve, 0.687). Conclusions: Dynamic model updating can improve model accuracy, discrimination, and calibration. The decision as to which method to use may depend on which measure is most important in each clinical context. Because competing therapies have emerged for valve replacement models, updating may guide clinical decision making.

Temporal validation and updating of a prediction model for the diagnosis of gestational diabetes mellitus.

OBJECTIVE: The original Monash gestational diabetes mellitus (GDM) risk prediction in early pregnancy model is internationally externally validated and clinically implemented. We temporally validate and update this model in a contemporary population with a universal screening context and revised diagnostic criteria and ethnicity categories, thereby improving model performance and generalizability. STUDY DESIGN AND SETTING: The updating dataset comprised of routinely collected health data for singleton pregnancies delivered in Melbourne, Australia from 2016 to 2018. Model predictors included age, body mass index, ethnicity, diabetes family history, GDM history, and poor obstetric outcome history. Model updating methods were recalibration-in-the-large (Model A), intercept and slope re-estimation (Model B), and coefficient revision using logistic regression (Model C1, original ethnicity categories; Model C2, revised ethnicity categories). Analysis included 10-fold cross-validation, assessment of performance measures (c-statistic, calibration-in-the-large, calibration slope, and expected-observed ratio), and a closed-loop testing procedure to compare models' log-likelihood and akaike information criterion scores. RESULTS: In 26,474 singleton pregnancies (4,756, 18% with GDM), the original model demonstrated reasonable temporal validation (c-statistic = 0.698) but suboptimal calibration (expected-observed ratio = 0.485). Updated model C2 was preferred, with a high c-statistic (0.732) and significantly better performance in closed testing. CONCLUSION: We demonstrated updating methods to sustain predictive performance in a contemporary population, highlighting the value and versatility of prediction models for guiding risk-stratified GDM care.

Integrating optical imaging techniques for a novel approach to evaluate Siberian wild rye seed maturity.

Advances in optical imaging technology using rapid and non-destructive methods have led to improvements in the efficiency of seed quality detection. Accurately timing the harvest is crucial for maximizing the yield of higher-quality Siberian wild rye seeds by minimizing excessive shattering during harvesting. This research applied integrated optical imaging techniques and machine learning algorithms to develop different models for classifying Siberian wild rye seeds based on different maturity stages and grain positions. The multi-source fusion of morphological, multispectral, and autofluorescence data provided more comprehensive information but also increases the performance requirements of the equipment. Therefore, we employed three filtering algorithms, namely minimal joint mutual information maximization (JMIM), information gain, and Gini impurity, and set up two control methods (feature union and no-filtering) to assess the impact of retaining only 20% of the features on the model performance. Both JMIM and information gain revealed autofluorescence and morphological features (CIELab A, CIELab B, hue and saturation), with these two filtering algorithms showing shorter run times. Furthermore, a strong correlation was observed between shoot length and morphological and autofluorescence spectral features. Machine learning models based on linear discriminant analysis (LDA), random forests (RF) and support vector machines (SVM) showed high performance (>0.78 accuracies) in classifying seeds at different maturity stages. Furthermore, it was found that there was considerable variation in the different grain positions at the maturity stage, and the K-means approach was used to improve the model performance by 5.8%-9.24%. In conclusion, our study demonstrated that feature filtering algorithms combined with machine learning algorithms offer high performance and low cost in identifying seed maturity stages and that the application of k-means techniques for inconsistent maturity improves classification accuracy. Therefore, this technique could be employed classification of seed maturity and superior physiological quality for Siberian wild rye seeds.

Clinical prediction models for serious infections in children: external validation in ambulatory care.

BACKGROUND: Early distinction between mild and serious infections (SI) is challenging in children in ambulatory care. Clinical prediction models (CPMs), developed to aid physicians in clinical decision-making, require broad external validation before clinical use. We aimed to externally validate four CPMs, developed in emergency departments, in ambulatory care. METHODS: We applied the CPMs in a prospective cohort of acutely ill children presenting to general practices, outpatient paediatric practices or emergency departments in Flanders, Belgium. For two multinomial regression models, Feverkidstool and Craig model, discriminative ability and calibration were assessed, and a model update was performed by re-estimation of coefficients with correction for overfitting. For two risk scores, the SBI score and PAWS, the diagnostic test accuracy was assessed. RESULTS: A total of 8211 children were included, comprising 498 SI and 276 serious bacterial infections (SBI). Feverkidstool had a C-statistic of 0.80 (95% confidence interval 0.77-0.84) with good calibration for pneumonia and 0.74 (0.70-0.79) with poor calibration for other SBI. The Craig model had a C-statistic of 0.80 (0.77-0.83) for pneumonia, 0.75 (0.70-0.80) for complicated urinary tract infections and 0.63 (0.39-0.88) for bacteraemia, with poor calibration. The model update resulted in improved C-statistics for all outcomes and good overall calibration for Feverkidstool and the Craig model. SBI score and PAWS performed extremely weak with sensitivities of 0.12 (0.09-0.15) and 0.32 (0.28-0.37). CONCLUSIONS: Feverkidstool and the Craig model show good discriminative ability for predicting SBI and a potential for early recognition of SBI, confirming good external validity in a low prevalence setting of SBI. The SBI score and PAWS showed poor diagnostic performance. TRIAL REGISTRATION: ClinicalTrials.gov, NCT02024282. Registered on 31 December 2013.

The Need to Prioritize Model-Updating Processes in Clinical Artificial Intelligence (AI) Models: Protocol for a Scoping Review.

BACKGROUND: With an increase in the number of artificial intelligence (AI) and machine learning (ML) algorithms available for clinical settings, appropriate model updating and implementation of updates are imperative to ensure applicability, reproducibility, and patient safety. OBJECTIVE: The objective of this scoping review was to evaluate and assess the model-updating practices of AI and ML clinical models that are used in direct patient-provider clinical decision-making. METHODS: We used the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist and the PRISMA-P protocol guidance in addition to a modified CHARMS (Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies) checklist to conduct this scoping review. A comprehensive medical literature search of databases, including Embase, MEDLINE, PsycINFO, Cochrane, Scopus, and Web of Science, was conducted to identify AI and ML algorithms that would impact clinical decision-making at the level of direct patient care. Our primary end point is the rate at which model updating is recommended by published algorithms; we will also conduct an assessment of study quality and risk of bias in all publications reviewed. In addition, we will evaluate the rate at which published algorithms include ethnic and gender demographic distribution information in their training data as a secondary end point. RESULTS: Our initial literature search yielded approximately 13,693 articles, with approximately 7810 articles to consider for full reviews among our team of 7 reviewers. We plan to complete the review process and disseminate the results by spring of 2023. CONCLUSIONS: Although AI and ML applications in health care have the potential to improve patient care by reducing errors between measurement and model output, currently there exists more hype than hope because of the lack of proper external validation of these models. We expect to find that the AI and ML model-updating methods are proxies for model applicability and generalizability on implementation. Our findings will add to the field by determining the degree to which published models meet the criteria for clinical validity, real-life implementation, and best practices to optimize model development, and in so doing, reduce the overpromise and underachievement of the contemporary model development process. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): PRR1-10.2196/37685.

Model Updating Concept Using Bridge Weigh-in-Motion Data.

Finite element (FE) model updating of bridges is based on the measured modal parameters and less frequently on the measured structural response under a known load. Until recently, the FE model updating did not consider strain measurements from sensors installed for weighing vehicles with bridge weigh-in-motion (B-WIM) systems. A 50-year-old multi-span concrete highway viaduct, renovated between 2017 and 2019, was equipped with continuous monitoring system with over 200 sensors, and a B-WIM system. In the most heavily instrumented span, the maximum measured longitudinal strains induced by the full-speed calibration vehicle passages were compared with the modelled strains. Based on the sensitivity study results, three variables that affected its overall stiffness were updated: Young's modulus adjustment factor of all structural elements, and two anchorage reduction factors that considered the interaction between the superstructure and non-structural elements. The analysis confirmed the importance of the initial manual FE model updating to correctly reflect the non-structural elements during the automatic nonlinear optimisation. It also demonstrated a successful use of pseudo-static B-WIM loading data during the model updating process and the potential to extend the proposed approach to using random B-WIM-weighed vehicles for FE model updating and long-term monitoring of structural parameters and load-dependent phenomena.

Development and External Validation of an Improved Version of the Diagnostic Model for Opportunistic Screening of Malignant Esophageal Lesions.

We aimed to develop an improved version of the diagnostic model predicting the risk of malignant esophageal lesions in opportunistic screening and validate it in external populations. The development set involved 10,595 outpatients receiving endoscopy from a hospital in Hua County, a high-risk region for esophageal squamous cell carcinoma in northern China. Validation set A enrolled 9453 outpatients receiving endoscopy in a non-high-risk region in southern China. Validation set B involved 17,511 residents in Hua County. The improved diagnostic model consisted of seven predictors including age, gender, family history of esophageal squamous cell carcinoma, smoking, body mass index, dysphagia, and retrosternal pain, with an area under the receiver operating characteristic curve (AUC) of 0.860 (95% confidence interval: 0.835-0.886) in the development set. Ideal discrimination ability was achieved in external validations (AUC validation set A: 0.892, 95% confidence interval: 0.858-0.926; AUC validation set B: 0.799, 95% confidence interval: 0.705-0.894). This improved model also markedly increased the detection rate of malignant esophageal lesions compared with universal screening, demonstrating great potential for use in opportunistic screening of malignant esophageal lesions in heterogeneous populations.

Probabilistic Structural Model Updating with Modal Flexibility Using a Modified Firefly Algorithm.

Structural model updating is one of the most important steps in structural health monitoring, which can achieve high-precision matching between finite element models and actual engineering structures. In this study, a Bayesian model updating method with modal flexibility was presented, where a modified heuristic optimization algorithm named modified Nelder-Mead firefly algorithm (m-NMFA) was proposed to find the most probable values (MPV) of model parameters for the maximum a posteriori probability (MAP) estimate. The proposed m-NMFA was compared to the original firefly algorithm (FA), the genetic algorithm (GA), and the particle swarm algorithm (PSO) through the numerical illustrative examples of 18 benchmark functions and a twelve-story shear frame model. Then, a six-story shear frame model test was performed to identify the inter-story stiffness of the structure in the original and the damage states, respectively. By comparing the two, the position and extent of damage were accurately found and quantified in a probabilistic manner. In terms of optimization, the proposed m-NMFA was powerful to find the MPVs much faster and more accurately. In the incomplete measurement case, only the m-NMFA achieved target damage identification results. The proposed Bayesian model updating method has the advantages of high precision, fast convergence, and strong robustness in MPV finding and the ability of parameter uncertainty quantification.

Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review.

BACKGROUND: Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models are fit for purpose and remain relevant in the long-term. We aimed to present an overview of methodological guidance for the evaluation (i.e., validation and impact assessment) and updating of clinical prediction models. METHODS: We systematically searched nine databases from January 2000 to January 2022 for articles in English with methodological recommendations for the post-derivation stages of interest. Qualitative analysis was used to summarize the 70 selected guidance papers. RESULTS: Key aspects for validation are the assessment of statistical performance using measures for discrimination (e.g., C-statistic) and calibration (e.g., calibration-in-the-large and calibration slope). For assessing impact or usefulness in clinical decision-making, recent papers advise using decision-analytic measures (e.g., the Net Benefit) over simplistic classification measures that ignore clinical consequences (e.g., accuracy, overall Net Reclassification Index). Commonly recommended methods for model updating are recalibration (i.e., adjustment of intercept or baseline hazard and/or slope), revision (i.e., re-estimation of individual predictor effects), and extension (i.e., addition of new markers). Additional methodological guidance is needed for newer types of updating (e.g., meta-model and dynamic updating) and machine learning-based models. CONCLUSION: Substantial guidance was found for model evaluation and more conventional updating of regression-based models. An important development in model evaluation is the introduction of a decision-analytic framework for assessing clinical usefulness. Consensus is emerging on methods for model updating.

Time-Aware Explainable Recommendation via Updating Enabled Online Prediction.

There has been growing attention on explainable recommendation that is able to provide high-quality results as well as intuitive explanations. However, most existing studies use offline prediction strategies where recommender systems are trained once while used forever, which ignores the dynamic and evolving nature of user-item interactions. There are two main issues with these methods. First, their random dataset split setting will result in data leakage that knowledge should not be known at the time of training is utilized. Second, the dynamic characteristics of user preferences are overlooked, resulting in a model aging issue where the model's performance degrades along with time. In this paper, we propose an updating enabled online prediction framework for the time-aware explainable recommendation. Specifically, we propose an online prediction scheme to eliminate the data leakage issue and two novel updating strategies to relieve the model aging issue. Moreover, we conduct extensive experiments on four real-world datasets to evaluate the effectiveness of our proposed methods. Compared with the state-of-the-art, our time-aware approach achieves higher accuracy results and more convincing explanations for the entire lifetime of recommendation systems, i.e., both the initial period and the long-term usage.

Development and validation of an early warning model for hospitalized COVID-19 patients: a multi-center retrospective cohort study.

BACKGROUND: Timely identification of deteriorating COVID-19 patients is needed to guide changes in clinical management and admission to intensive care units (ICUs). There is significant concern that widely used Early warning scores (EWSs) underestimate illness severity in COVID-19 patients and therefore, we developed an early warning model specifically for COVID-19 patients. METHODS: We retrospectively collected electronic medical record data to extract predictors and used these to fit a random forest model. To simulate the situation in which the model would have been developed after the first and implemented during the second COVID-19 'wave' in the Netherlands, we performed a temporal validation by splitting all included patients into groups admitted before and after August 1, 2020. Furthermore, we propose a method for dynamic model updating to retain model performance over time. We evaluated model discrimination and calibration, performed a decision curve analysis, and quantified the importance of predictors using SHapley Additive exPlanations values. RESULTS: We included 3514 COVID-19 patient admissions from six Dutch hospitals between February 2020 and May 2021, and included a total of 18 predictors for model fitting. The model showed a higher discriminative performance in terms of partial area under the receiver operating characteristic curve (0.82 [0.80-0.84]) compared to the National early warning score (0.72 [0.69-0.74]) and the Modified early warning score (0.67 [0.65-0.69]), a greater net benefit over a range of clinically relevant model thresholds, and relatively good calibration (intercept = 0.03 [- 0.09 to 0.14], slope = 0.79 [0.73-0.86]). CONCLUSIONS: This study shows the potential benefit of moving from early warning models for the general inpatient population to models for specific patient groups. Further (independent) validation of the model is needed.