Detecting dental caries on oral photographs using artificial intelligence: A systematic review.

OBJECTIVES: This systematic review aimed at evaluating the performance of artificial intelligence (AI) models in detecting dental caries on oral photographs. METHODS: Methodological characteristics and performance metrics of clinical studies reporting on deep learning and other machine learning algorithms were assessed. The risk of bias was evaluated using the quality assessment of diagnostic accuracy studies 2 (QUADAS-2) tool. A systematic search was conducted in EMBASE, Medline, and Scopus. RESULTS: Out of 3410 identified records, 19 studies were included with six and seven studies having low risk of biases and applicability concerns for all the domains, respectively. Metrics varied widely and were assessed on multiple levels. F1-scores for classification and detection tasks were 68.3%-94.3% and 42.8%-95.4%, respectively. Irrespective of the task, F1-scores were 68.3%-95.4% for professional cameras, 78.8%-87.6%, for intraoral cameras, and 42.8%-80% for smartphone cameras. Limited studies allowed assessing AI performance for lesions of different severity. CONCLUSION: Automatic detection of dental caries using AI may provide objective verification of clinicians' diagnoses and facilitate patient-clinician communication and teledentistry. Future studies should consider more robust study designs, employ comparable and standardized metrics, and focus on the severity of caries lesions.

The Global Open Source Severity of Illness Score (GOSSIS).

OBJECTIVES: To develop and demonstrate the feasibility of a Global Open Source Severity of Illness Score (GOSSIS)-1 for critical care patients, which generalizes across healthcare systems and countries. DESIGN: A merger of several critical care multicenter cohorts derived from registry and electronic health record data. Data were split into training (70%) and test (30%) sets, using each set exclusively for development and evaluation, respectively. Missing data were imputed when not available. SETTING/PATIENTS: Two large multicenter datasets from Australia and New Zealand (Australian and New Zealand Intensive Care Society Adult Patient Database [ANZICS-APD]) and the United States (eICU Collaborative Research Database [eICU-CRD]) representing 249,229 and 131,051 patients, respectively. ANZICS-APD and eICU-CRD contributed data from 162 and 204 hospitals, respectively. The cohort included all ICU admissions discharged in 2014-2015, excluding patients less than 16 years old, admissions less than 6 hours, and those with a previous ICU stay. INTERVENTIONS: Not applicable. MEASUREMENTS AND MAIN RESULTS: GOSSIS-1 uses data collected during the ICU stay's first 24 hours, including extrema values for vital signs and laboratory results, admission diagnosis, the Glasgow Coma Scale, chronic comorbidities, and admission/demographic variables. The datasets showed significant variation in admission-related variables, case-mix, and average physiologic state. Despite this heterogeneity, test set discrimination of GOSSIS-1 was high (area under the receiver operator characteristic curve [AUROC], 0.918; 95% CI, 0.915-0.921) and calibration was excellent (standardized mortality ratio [SMR], 0.986; 95% CI, 0.966-1.005; Brier score, 0.050). Performance was held within ANZICS-APD (AUROC, 0.925; SMR, 0.982; Brier score, 0.047) and eICU-CRD (AUROC, 0.904; SMR, 0.992; Brier score, 0.055). Compared with GOSSIS-1, Acute Physiology and Chronic Health Evaluation (APACHE)-IIIj (ANZICS-APD) and APACHE-IVa (eICU-CRD), had worse discrimination with AUROCs of 0.904 and 0.869, and poorer calibration with SMRs of 0.594 and 0.770, and Brier scores of 0.059 and 0.063, respectively. CONCLUSIONS: GOSSIS-1 is a modern, free, open-source inhospital mortality prediction algorithm for critical care patients, achieving excellent discrimination and calibration across three countries.

An Evaluation of the Influence of Body Mass Index on Severity Scoring.

OBJECTIVES: Although one third or more of critically ill patients in the United States are obese, obesity is not incorporated as a contributing factor in any of the commonly used severity of illness scores. We hypothesize that selected severity of illness scores would perform differently if body mass index categorization was incorporated and that the performance of these score models would improve after consideration of body mass index as an additional model feature. DESIGN: Retrospective cohort analysis from a multicenter ICU database which contains deidentified data for more than 200,000 ICU admissions from 208 distinct ICUs across the United States between 2014 and 2015. SETTING: First ICU admission of patients with documented height and weight. PATIENTS: One-hundred eight-thousand four-hundred two patients from 189 different ICUs across United States were included in the analyses, of whom 4,661 (4%) were classified as underweight, 32,134 (30%) as normal weight, 32,278 (30%) as overweight, 30,259 (28%) as obese, and 9,070 (8%) as morbidly obese. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: To assess the effect of adding body mass index as a risk adjustment element to the Acute Physiology and Chronic Health Evaluation IV and Oxford Acute Severity of Illness scoring systems, we examined the impact of this addition on both discrimination and calibration. We performed three assessments based upon 1) the original scoring systems, 2) a recalibrated version of the systems, and 3) a recalibrated version incorporating body mass index as a covariate. We also performed a subgroup analysis in groups defined using World Health Organization guidelines for obesity. Incorporating body mass index into the models provided a minor improvement in both discrimination and calibration. In a subgroup analysis, model discrimination was higher in groups with higher body mass index, but calibration worsened. CONCLUSIONS: The performance of ICU prognostic models utilizing body mass index category as a scoring element was inconsistent across body mass index categories. Overall, adding body mass index as a risk adjustment variable led only to a minor improvement in scoring system performance.

Prolonged Elevated Heart Rate and 90-Day Survival in Acutely Ill Patients: Data From the MIMIC-III Database.

PURPOSE: We sought to evaluate the association of prolonged elevated heart rate (peHR) with survival in acutely ill patients. METHODS: We used a large observational intensive care unit (ICU) database (Multiparameter Intelligent Monitoring in Intensive Care III [MIMIC-III]), where frequent heart rate measurements were available. The peHR was defined as a heart rate >100 beats/min in 11 of 12 consecutive hours. The outcome was survival status at 90 days. We collected heart rates, disease severity (simplified acute physiology scores [SAPS II]), comorbidities (Charlson scores), and International Classification of Diseases (ICD) diagnosis information in 31 513 patients from the MIMIC-III ICU database. Propensity score (PS) methods followed by inverse probability weighting based on the PS was used to balance the 2 groups (the presence/absence of peHR). Multivariable weighted logistic regression was used to assess for association of peHR with the outcome survival at 90 days adjusting for additional covariates. RESULTS: The mean age was 64 years, and the most frequent main disease category was circulatory disease (41%). The mean SAPS II score was 35, and the mean Charlson comorbidity score was 2.3. Overall survival of the cohort at 90 days was 82%. Adjusted logistic regression showed a significantly increased risk of death within 90 days in patients with an episode of peHR (P < .001; odds ratio for death 1.79; confidence interval, 1.69-1.88). This finding was independent of median heart rate. CONCLUSION: We found a significant association of peHR with decreased survival in a large and heterogenous cohort of ICU patients.

Impact of Intensive Care Unit Discharge Delays on Patient Outcomes: A Retrospective Cohort Study.

OBJECTIVE: Patients often overstay in intensive care units (ICU) after they are deemed discharge ready. The objective of this study was to examine the impact of such discharge delays (DD) on subsequent in-hospital morbidity and mortality. DESIGN: Retrospective cohort study. SETTING: Single tertiary academic medical center. PATIENTS: Adult patients admitted to the medical ICU between 2005 and 2011. INTERVENTIONS: For all patients, DD (ie, time between request for a ward bed and time of ICU discharge) was calculated. Discharge delays was dichotomized as long (≥24 hours) or short (<24 hours). Multivariable linear and logistic regressions were used to assess the association between dichotomized DD and post-ICU clinical outcomes. RESULTS: Overall, 9673 discharges were included of which 10.4% patients had long DDs. In the fully adjusted model, a long delay was not associated with increased odds of death (adjusted odds ratio [aOR]: 0.99, 95% confidence interval [CI]: 0.74-1.31, P = .95) but was associated with a shorter log plus one of post-ICU discharge length of stay (LOS; regression coefficient: -0.13, 95% CI: -0.17 to -0.08, P < .001). Longer DD was not associated with more hospital-free days (HFD: a composite of post-ICU LOS and in-hospital mortality). Shorter DDs were associated with shorter LOS when LOS was measured from the time of ward bed request as opposed to the time of ICU discharge. CONCLUSIONS: In this study, long DD was associated with a slight decrease in post-ICU LOS but longer LOS when measured from the point of ward bed request, suggesting a potential role for more aggressive discharge planning in the ICU for patients with long DDs. There was no association between long DD and subsequent mortality or HFD.

Severity of Illness Scores May Misclassify Critically Ill Obese Patients.

OBJECTIVE: Severity of illness scores rest on the assumption that patients have normal physiologic values at baseline and that patients with similar severity of illness scores have the same degree of deviation from their usual state. Prior studies have reported differences in baseline physiology, including laboratory markers, between obese and normal weight individuals, but these differences have not been analyzed in the ICU. We compared deviation from baseline of pertinent ICU laboratory test results between obese and normal weight patients, adjusted for the severity of illness. DESIGN: Retrospective cohort study in a large ICU database. SETTING: Tertiary teaching hospital. PATIENTS: Obese and normal weight patients who had laboratory results documented between 3 days and 1 year prior to hospital admission. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Seven hundred sixty-nine normal weight patients were compared with 1,258 obese patients. After adjusting for the severity of illness score, age, comorbidity index, baseline laboratory result, and ICU type, the following deviations were found to be statistically significant: WBC 0.80 (95% CI, 0.27-1.33) × 10/L; p = 0.003; log (blood urea nitrogen) 0.01 (95% CI, 0.00-0.02); p = 0.014; log (creatinine) 0.03 (95% CI, 0.02-0.05), p < 0.001; with all deviations higher in obese patients. A logistic regression analysis suggested that after adjusting for age and severity of illness at least one of these deviations had a statistically significant effect on hospital mortality (p = 0.009). CONCLUSIONS: Among patients with the same severity of illness score, we detected clinically small but significant deviations in WBC, creatinine, and blood urea nitrogen from baseline in obese compared with normal weight patients. These small deviations are likely to be increasingly important as bigger data are analyzed in increasingly precise ways. Recognition of the extent to which all critically ill patients may deviate from their own baseline may improve the objectivity, precision, and generalizability of ICU mortality prediction and severity adjustment models.

A Comparative Analysis of Sepsis Identification Methods in an Electronic Database.

OBJECTIVES: To evaluate the relative validity of criteria for the identification of sepsis in an ICU database. DESIGN: Retrospective cohort study of adult ICU admissions from 2008 to 2012. SETTING: Tertiary teaching hospital in Boston, MA. PATIENTS: Initial admission of all adult patients to noncardiac surgical ICUs. INTERVENTIONS: Comparison of five different algorithms for retrospectively identifying sepsis, including the Sepsis-3 criteria. MEASUREMENTS AND MAIN RESULTS: 11,791 of 23,620 ICU admissions (49.9%) met criteria for the study. Within this subgroup, 59.9% were suspected of infection on ICU admission, 75.2% of admissions had Sequential Organ Failure Assessment greater than or equal to 2, and 49.1% had both suspicion of infection and Sequential Organ Failure Assessment greater than or equal to 2 thereby meeting the Sepsis-3 criteria. The area under the receiver operator characteristic of Sequential Organ Failure Assessment (0.74) for hospital mortality was consistent with previous studies of the Sepsis-3 criteria. The Centers for Disease Control and Prevention, Angus, Martin, Centers for Medicare & Medicaid Services, and explicit coding methods for identifying sepsis revealed respective sepsis incidences of 31.9%, 28.6%, 14.7%, 11.0%, and 9.0%. In-hospital mortality increased with decreasing cohort size, ranging from 30.1% (explicit codes) to 14.5% (Sepsis-3 criteria). Agreement among the criteria was acceptable (Cronbach's alpha, 0.40-0.62). CONCLUSIONS: The new organ dysfunction-based Sepsis-3 criteria have been proposed as a clinical method for identifying sepsis. These criteria identified a larger, less severely ill cohort than that identified by previously used administrative definitions. The Sepsis-3 criteria have several advantages over prior methods, including less susceptibility to coding practices changes, provision of temporal context, and possession of high construct validity. However, the Sepsis-3 criteria also present new challenges, especially when calculated retrospectively. Future studies on sepsis should recognize the differences in outcome incidence among identification methods and contextualize their findings according to the different cohorts identified.

Machine Learning and Decision Support in Critical Care.

Clinical data management systems typically provide caregiver teams with useful information, derived from large, sometimes highly heterogeneous, data sources that are often changing dynamically. Over the last decade there has been a significant surge in interest in using these data sources, from simply re-using the standard clinical databases for event prediction or decision support, to including dynamic and patient-specific information into clinical monitoring and prediction problems. However, in most cases, commercial clinical databases have been designed to document clinical activity for reporting, liability and billing reasons, rather than for developing new algorithms. With increasing excitement surrounding "secondary use of medical records" and "Big Data" analytics, it is important to understand the limitations of current databases and what needs to change in order to enter an era of "precision medicine." This review article covers many of the issues involved in the collection and preprocessing of critical care data. The three challenges in critical care are considered: compartmentalization, corruption, and complexity. A range of applications addressing these issues are covered, including the modernization of static acuity scoring; on-line patient tracking; personalized prediction and risk assessment; artifact detection; state estimation; and incorporation of multimodal data sources such as genomic and free text data.

Prognosing post-treatment outcomes of head and neck cancer using structured data and machine learning: A systematic review.

BACKGROUND: This systematic review aimed to evaluate the performance of machine learning (ML) models in predicting post-treatment survival and disease progression outcomes, including recurrence and metastasis, in head and neck cancer (HNC) using clinicopathological structured data. METHODS: A systematic search was conducted across the Medline, Scopus, Embase, Web of Science, and Google Scholar databases. The methodological characteristics and performance metrics of studies that developed and validated ML models were assessed. The risk of bias was evaluated using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). RESULTS: Out of 5,560 unique records, 34 articles were included. For survival outcome, the ML model outperformed the Cox proportional hazards model in time-to-event analyses for HNC, with a concordance index of 0.70-0.79 vs. 0.66-0.76, and for all sub-sites including oral cavity (0.73-0.89 vs. 0.69-0.77) and larynx (0.71-0.85 vs. 0.57-0.74). In binary classification analysis, the area under the receiver operating characteristics (AUROC) of ML models ranged from 0.75-0.97, with an F1-score of 0.65-0.89 for HNC; AUROC of 0.61-0.91 and F1-score of 0.58-0.86 for the oral cavity; and AUROC of 0.76-0.97 and F1-score of 0.63-0.92 for the larynx. Disease-specific survival outcomes showed higher performance than overall survival outcomes, but the performance of ML models did not differ between three- and five-year follow-up durations. For disease progression outcomes, no time-to-event metrics were reported for ML models. For binary classification of the oral cavity, the only evaluated subsite, the AUROC ranged from 0.67 to 0.97, with F1-scores between 0.53 and 0.89. CONCLUSIONS: ML models have demonstrated considerable potential in predicting post-treatment survival and disease progression, consistently outperforming traditional linear models and their derived nomograms. Future research should incorporate more comprehensive treatment features, emphasize disease progression outcomes, and establish model generalizability through external validations and the use of multicenter datasets.

Predicting non-muscle invasive bladder cancer outcomes using artificial intelligence: a systematic review using APPRAISE-AI.

Accurate prediction of recurrence and progression in non-muscle invasive bladder cancer (NMIBC) is essential to inform management and eligibility for clinical trials. Despite substantial interest in developing artificial intelligence (AI) applications in NMIBC, their clinical readiness remains unclear. This systematic review aimed to critically appraise AI studies predicting NMIBC outcomes, and to identify common methodological and reporting pitfalls. MEDLINE, EMBASE, Web of Science, and Scopus were searched from inception to February 5th, 2024 for AI studies predicting NMIBC recurrence or progression. APPRAISE-AI was used to assess methodological and reporting quality of these studies. Performance between AI and non-AI approaches included within these studies were compared. A total of 15 studies (five on recurrence, four on progression, and six on both) were included. All studies were retrospective, with a median follow-up of 71 months (IQR 32-93) and median cohort size of 125 (IQR 93-309). Most studies were low quality, with only one classified as high quality. While AI models generally outperformed non-AI approaches with respect to accuracy, c-index, sensitivity, and specificity, this margin of benefit varied with study quality (median absolute performance difference was 10 for low, 22 for moderate, and 4 for high quality studies). Common pitfalls included dataset limitations, heterogeneous outcome definitions, methodological flaws, suboptimal model evaluation, and reproducibility issues. Recommendations to address these challenges are proposed. These findings emphasise the need for collaborative efforts between urological and AI communities paired with rigorous methodologies to develop higher quality models, enabling AI to reach its potential in enhancing NMIBC care.

Deep neural networks detect regional wall motion abnormalities and preclinical cardiovascular disease from 12-lead ECGs.

Background: Identifying regional wall motion abnormalities (RWMAs) is critical for diagnosing and risk stratifying patients with cardiovascular disease, particularly ischemic heart disease. We hypothesized that a deep neural network could accurately identify patients with regional wall motion abnormalities from a readily available standard 12-lead electrocardiogram (ECG). Methods: This observational, retrospective study included patients who were treated at Beth Israel Deaconess Medical Center and had an ECG and echocardiogram performed within 14 days of each other between 2008 and 2019. We trained a convolutional neural network to detect the presence of RWMAs, qualitative global right ventricular (RV) hypokinesis, and varying degrees of left ventricular dysfunction (left ventricular ejection fraction [LVEF] ≤50%, LVEF ≤40%, and LVEF ≤35%) identified by echocardiography, using ECG data alone. Patients were randomly split into development (80%) and test sets (20%). Model performance was assessed using area under the receiver operating characteristic curve (AUC). Cox proportional hazard models adjusted for age and sex were performed to estimate the risk of future acute coronary events. Results: The development set consisted of 19,837 patients (mean age 66.7±16.4; 46.7% female) and the test set comprised of 4,953 patients (mean age 67.5±15.8 years; 46.5% female). On the test dataset, the model accurately identified the presence of RWMA, RV hypokinesis, LVEF ≤50%, LVEF ≤40%, and LVEF ≤35% with AUCs of 0.87 (95% CI 0.858-0.882), 0.888 (95% CI 0.878-0.899), 0.923 (95% CI 0.914-0.933), 0.93 (95% CI 0.921-0.939), and 0.876 (95% CI 0.858-0.896), respectively. Among patients with normal biventricular function at the time of the index ECG, those classified as having RMWA by the model were 3 times the risk (age- and sex-adjusted hazard ratio, 2.8; 95% CI 1.9-3.9) for future acute coronary events compared to those classified as negative. Conclusions: We demonstrate that a deep neural network can help identify regional wall motion abnormalities and reduced LV function from a 12-lead ECG and could potentially be used as a screening tool for triaging patients who need either initial or repeat echocardiographic imaging.

A new severity of illness scale using a subset of Acute Physiology And Chronic Health Evaluation data elements shows comparable predictive accuracy.

OBJECTIVES: Severity of illness scores have gained considerable interest for their use in predicting outcomes such as mortality and length of stay. The most sophisticated scoring systems require the collection of numerous physiologic measurements, making their use in real-time difficult. A severity of illness score based on a few parameters that can be captured electronically would be of great benefit. Using a machine-learning technique known as particle swarm optimization, we attempted to reduce the number of physiologic parameters collected in the Acute Physiology, Age, and Chronic Health Evaluation IV system without losing predictive accuracy. DESIGN: Retrospective cohort study of ICU admissions from 2007 to 2011. SETTING: Eighty-six ICUs at 49 U.S. hospitals where an Acute Physiology, Age, and Chronic Health Evaluation IV system had been installed. PATIENTS: 81,087 admissions, of which 72,474 did not have any missing values. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Machine-learning algorithms were used to come up with the minimal set of variables that were capable of yielding an accurate severity of illness score: the Oxford Acute Severity of Illness Score. Predictive models of ICU mortality using Oxford Acute Severity of Illness Score were developed on admissions during 2007-2009 and validated on admissions during 2010-2011. The most parsimonious Oxford Acute Severity of Illness Score consisted of seven physiologic measurements, elective surgery, age, and prior length of stay. Predictive models of ICU mortality using Oxford Acute Severity of Illness Score achieved an area under the receiver operating characteristic curve of 0.88 and calibrated well. CONCLUSIONS: A reduced severity of illness score had discrimination and calibration equivalent to more complex existing models. This was accomplished in large part using machine-learning algorithms, which can effectively account for the nonlinear associations between physiologic parameters and outcome.

AnnoDash, a clinical terminology annotation dashboard.

Background: Standard ontologies are critical for interoperability and multisite analyses of health data. Nevertheless, mapping concepts to ontologies is often done with generic tools and is labor-intensive. Contextualizing candidate concepts within source data is also done in an ad hoc manner. Methods and Results: We present AnnoDash, a flexible dashboard to support annotation of concepts with terms from a given ontology. Text-based similarity is used to identify likely matches, and large language models are used to improve ontology ranking. A convenient interface is provided to visualize observations associated with a concept, supporting the disambiguation of vague concept descriptions. Time-series plots contrast the concept with known clinical measurements. We evaluated the dashboard qualitatively against several ontologies (SNOMED CT, LOINC, etc.) by using MIMIC-IV measurements. The dashboard is web-based and step-by-step instructions for deployment are provided, simplifying usage for nontechnical audiences. The modular code structure enables users to extend upon components, including improving similarity scoring, constructing new plots, or configuring new ontologies. Conclusion: AnnoDash, an improved clinical terminology annotation tool, can facilitate data harmonizing by promoting mapping of clinical data. AnnoDash is freely available at https://github.com/justin13601/AnnoDash (https://doi.org/10.5281/zenodo.8043943).

MIMIC-IV on FHIR: converting a decade of in-patient data into an exchangeable, interoperable format.

OBJECTIVE: Convert the Medical Information Mart for Intensive Care (MIMIC)-IV database into Health Level 7 Fast Healthcare Interoperability Resources (FHIR). Additionally, generate and publish an openly available demo of the resources, and create a FHIR Implementation Guide to support and clarify the usage of MIMIC-IV on FHIR. MATERIALS AND METHODS: FHIR profiles and terminology system of MIMIC-IV were modeled from the base FHIR R4 resources. Data and terminology were reorganized from the relational structure into FHIR according to the profiles. Resources generated were validated for conformance with the FHIR profiles. Finally, FHIR resources were published as newline delimited JSON files and the profiles were packaged into an implementation guide. RESULTS: The modeling of MIMIC-IV in FHIR resulted in 25 profiles, 2 extensions, 35 ValueSets, and 34 CodeSystems. An implementation guide encompassing the FHIR modeling can be accessed at mimic.mit.edu/fhir/mimic. The generated demo dataset contained 100 patients and over 915 000 resources. The full dataset contained 315 000 patients covering approximately 5 840 000 resources. The final datasets in NDJSON format are accessible on PhysioNet. DISCUSSION: Our work highlights the challenges and benefits of generating a real-world FHIR store. The challenges arise from terminology mapping and profiling modeling decisions. The benefits come from the extensively validated openly accessible data created as a result of the modeling work. CONCLUSION: The newly created MIMIC-IV on FHIR provides one of the first accessible deidentified critical care FHIR datasets. The extensive real-world data found in MIMIC-IV on FHIR will be invaluable for research and the development of healthcare applications.

Evaluation of evolving sepsis screening criteria in discriminating suspected sepsis and mortality among adult patients admitted to the intensive care unit.

BACKGROUND: Institutions struggle with successful use of sepsis alerts within electronic health records. OBJECTIVE: Test the association of sepsis screening measurement criteria in discrimination of mortality and detection of sepsis in a large dataset. DESIGN: Retrospective, cohort study using a large United States (U.S.) intensive care database. The Institutional Review Board exempt status was obtained from Kansas University Medical Center Human Research Protection Program (10-1-2015). SETTING: 334 U.S. hospitals participating in the eICU Research Institute. PARTICIPANTS: Nine hundred twelve thousand five hundred and nine adult intensive care admissions from 183 hospitals. METHODS: Exposures included: systemic inflammatory response syndrome criteria ≥ 2 (Sepsis-1); systemic inflammatory response syndrome criteria with organ failure criteria ≥ 3.5 points (Sepsis-2); and sepsis-related organ failure assessment score ≥ 2 and quick score ≥ 2 (Sepsis-3). Discrimination of outcomes was determined with/without (adjusted/unadjusted) baseline risk exposure to a model. The receiver operating characteristic curve (AUROC) and odds ratios (ORs) for each decile of baseline risk of sepsis or death were assessed. RESULTS: Within the eligible cohort of 912,509, a total of 86,219 (9.4 %) patients did not survive their hospital stay and 186,870 (20.5 %) met the definition of suspected sepsis. For suspected sepsis discrimination, Sepsis-2 (unadjusted AUROC 0.67, 99 % CI: 0.66-0.67 and adjusted AUROC 0.77, 99 % CI: 0.77-0.77) outperformed Sepsis-3 (SOFA unadjusted AUROC 0.61, 99 % CI: 0.61-0.61 and adjusted AUROC 0.74, 99 % CI: 0.74-0.74) (qSOFA unadjusted AUROC 0.59, 99 % CI: 0.59-0.60 and adjusted AUROC 0.73, 99 % CI: 0.73-0.73). Sepsis-2 also outperformed Sepsis-1 (unadjusted AUROC 0.58, 99 % CI: 0.58-0.58 and adjusted AUROC 0.73, 99 % CI: 0.73-0.73). In between differences of AUROCs were statistically significantly different. Sepsis-2 ORs were higher for the outcome of suspected sepsis when considering deciles of risk than the other measurement systems. CONCLUSIONS AND RELEVANCE: Sepsis-2 outperformed other systems in suspected sepsis detection and was comparable to SOFA in prognostic accuracy of mortality in adult intensive care patients.

APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support.

Importance: Artificial intelligence (AI) has gained considerable attention in health care, yet concerns have been raised around appropriate methods and fairness. Current AI reporting guidelines do not provide a means of quantifying overall quality of AI research, limiting their ability to compare models addressing the same clinical question. Objective: To develop a tool (APPRAISE-AI) to evaluate the methodological and reporting quality of AI prediction models for clinical decision support. Design, Setting, and Participants: This quality improvement study evaluated AI studies in the model development, silent, and clinical trial phases using the APPRAISE-AI tool, a quantitative method for evaluating quality of AI studies across 6 domains: clinical relevance, data quality, methodological conduct, robustness of results, reporting quality, and reproducibility. These domains included 24 items with a maximum overall score of 100 points. Points were assigned to each item, with higher points indicating stronger methodological or reporting quality. The tool was applied to a systematic review on machine learning to estimate sepsis that included articles published until September 13, 2019. Data analysis was performed from September to December 2022. Main Outcomes and Measures: The primary outcomes were interrater and intrarater reliability and the correlation between APPRAISE-AI scores and expert scores, 3-year citation rate, number of Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) low risk-of-bias domains, and overall adherence to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement. Results: A total of 28 studies were included. Overall APPRAISE-AI scores ranged from 33 (low quality) to 67 (high quality). Most studies were moderate quality. The 5 lowest scoring items included source of data, sample size calculation, bias assessment, error analysis, and transparency. Overall APPRAISE-AI scores were associated with expert scores (Spearman ρ, 0.82; 95% CI, 0.64-0.91; P < .001), 3-year citation rate (Spearman ρ, 0.69; 95% CI, 0.43-0.85; P < .001), number of QUADAS-2 low risk-of-bias domains (Spearman ρ, 0.56; 95% CI, 0.24-0.77; P = .002), and adherence to the TRIPOD statement (Spearman ρ, 0.87; 95% CI, 0.73-0.94; P < .001). Intraclass correlation coefficient ranges for interrater and intrarater reliability were 0.74 to 1.00 for individual items, 0.81 to 0.99 for individual domains, and 0.91 to 0.98 for overall scores. Conclusions and Relevance: In this quality improvement study, APPRAISE-AI demonstrated strong interrater and intrarater reliability and correlated well with several study quality measures. This tool may provide a quantitative approach for investigators, reviewers, editors, and funding organizations to compare the research quality across AI studies for clinical decision support.