Real-Time Machine Learning Alerts to Prevent Escalation of Care: A Nonrandomized Clustered Pragmatic Clinical Trial.

OBJECTIVES: Machine learning algorithms can outperform older methods in predicting clinical deterioration, but rigorous prospective data on their real-world efficacy are limited. We hypothesized that real-time machine learning generated alerts sent directly to front-line providers would reduce escalations. DESIGN: Single-center prospective pragmatic nonrandomized clustered clinical trial. SETTING: Academic tertiary care medical center. PATIENTS: Adult patients admitted to four medical-surgical units. Assignment to intervention or control arms was determined by initial unit admission. INTERVENTIONS: Real-time alerts stratified according to predicted likelihood of deterioration sent either to the primary team or directly to the rapid response team (RRT). Clinical care and interventions were at the providers' discretion. For the control units, alerts were generated but not sent, and standard RRT activation criteria were used. MEASUREMENTS AND MAIN RESULTS: The primary outcome was the rate of escalation per 1000 patient bed days. Secondary outcomes included the frequency of orders for fluids, medications, and diagnostic tests, and combined in-hospital and 30-day mortality. Propensity score modeling with stabilized inverse probability of treatment weight (IPTW) was used to account for differences between groups. Data from 2740 patients enrolled between July 2019 and March 2020 were analyzed (1488 intervention, 1252 control). Average age was 66.3 years and 1428 participants (52%) were female. The rate of escalation was 12.3 vs. 11.3 per 1000 patient bed days (difference, 1.0; 95% CI, -2.8 to 4.7) and IPTW adjusted incidence rate ratio 1.43 (95% CI, 1.16-1.78; p < 0.001). Patients in the intervention group were more likely to receive cardiovascular medication orders (16.1% vs. 11.3%; 4.7%; 95% CI, 2.1-7.4%) and IPTW adjusted relative risk (RR) (1.74; 95% CI, 1.39-2.18; p < 0.001). Combined in-hospital and 30-day-mortality was lower in the intervention group (7% vs. 9.3%; -2.4%; 95% CI, -4.5% to -0.2%) and IPTW adjusted RR (0.76; 95% CI, 0.58-0.99; p = 0.045). CONCLUSIONS: Real-time machine learning alerts do not reduce the rate of escalation but may reduce mortality.

Systematic Review and Meta-analysis of the Prevalence of Levator Ani Avulsion with Obstetric Anal Sphincter Injury and its Effects on Pelvic Floor Dysfunction.

INTRODUCTION AND HYPOTHESIS: The objective was to evaluate the prevalence of levator ani avulsion (LAA) among primiparous women with obstetric anal sphincter injury (OASI) and how this association could affect future pelvic floor dysfunction. METHODS: Three electronic databases (MEDLINE/PubMed/EMBASE) were searched in December 2018 and again in October 2022. Nine full-text articles were included in the analysis. The exclusion criteria were language other than English, studies not based on primiparous women only, conference abstracts, and evaluation without ultrasound or MRI. RESULTS: The overall prevalence of LAA was 24% (95% CI: 18-30%). Those with OASI, were at a higher risk of LAA, OR 3.49 (95% CI: 1.46 to 8.35). In women with LAA + OASI versus OASI alone, Three of Five studies showed worsened AI symptoms. Three of Five studies assessing urinary incontinence (UI) reported no significant difference in UI, whereas two reported increased UI. All studies that looked at pelvic organ prolapse reported a higher incidence of symptomatic prolapse and reduced pelvic floor muscle strength in women with LAA + OASI compared with those without LAA. CONCLUSION: Levator ani avulsion is prevalent following vaginal birth and is strongly associated with OASI. Incidence of AI does not increase in women with LAA and OASI, but they had greater symptom bother. OASI with LAA appears to increase the incidence of pelvic floor weakness and pelvic organ prolapse. There is no consensus agreement on the effect of LAA + OASI on UI.

Investigating turbulence distribution in the lower atmosphere using time-lapse imagery from a camera bank.

The atmosphere's surface layer (first 50-100 m above the ground) is extremely dynamic and is influenced by surface radiative properties, roughness, and atmospheric stability. Understanding the distribution of turbulence in the surface layer is critical to many applications, such as directed energy and free space optical communications. Several measurement campaigns in the past have relied on weather balloons or sonic detection and ranging (SODAR) to measure turbulence up to the atmospheric boundary layer. However, these campaigns had limited measurements near the surface. We have developed a time-lapse imaging technique to profile atmospheric turbulence from turbulence-induced differential motion or tilts between features on a distant target, sensed between pairs of cameras in a camera bank. This is a low-cost and portable approach to remotely sense turbulence from a single site without the deployment of sensors at the target location. It is thus an excellent approach to study the distribution of turbulence in low altitudes with sufficiently high resolution. In the present work, the potential of this technique was demonstrated. We tested the method over a path with constant turbulence. We explored the turbulence distribution with height in the first 20 m above the ground by imaging a 30 m water tower over a flat terrain on three clear days in summer. In addition, we analyzed time-lapse data from a second water tower over a sloped terrain. In most of the turbulence profiles extracted from these images, the drop in turbulence with altitude in the first 15 m or so above the ground showed a h m dependence, where the exponent m varied from -0.3 to -1.0, quite contrary to the widely used value of -4/3.

Malnutrition risk assessment using a machine learning-based screening tool: A multicentre retrospective cohort.

BACKGROUND: Malnutrition is associated with increased morbidity, mortality, and healthcare costs. Early detection is important for timely intervention. This paper assesses the ability of a machine learning screening tool (MUST-Plus) implemented in registered dietitian (RD) workflow to identify malnourished patients early in the hospital stay and to improve the diagnosis and documentation rate of malnutrition. METHODS: This retrospective cohort study was conducted in a large, urban health system in New York City comprising six hospitals serving a diverse patient population. The study included all patients aged ≥ 18 years, who were not admitted for COVID-19 and had a length of stay of ≤ 30 days. RESULTS: Of the 7736 hospitalisations that met the inclusion criteria, 1947 (25.2%) were identified as being malnourished by MUST-Plus-assisted RD evaluations. The lag between admission and diagnosis improved with MUST-Plus implementation. The usability of the tool output by RDs exceeded 90%, showing good acceptance by users. When compared pre-/post-implementation, the rate of both diagnoses and documentation of malnutrition showed improvement. CONCLUSION: MUST-Plus, a machine learning-based screening tool, shows great promise as a malnutrition screening tool for hospitalised patients when used in conjunction with adequate RD staffing and training about the tool. It performed well across multiple measures and settings. Other health systems can use their electronic health record data to develop, test and implement similar machine learning-based processes to improve malnutrition screening and facilitate timely intervention.

Children and adolescents with all forms of shoulder instability demonstrate differences in their movement and muscle activity patterns when compared to age- and sex-matched controls.

HYPOTHESIS AND BACKGROUND: Shoulder instability (SI) is a complex impairment, and identifying biomarkers that differentiate subgroups is challenging. Children and adolescents with SI (irrespective of etiology) have differences in their movement and muscle activity profiles compared to age- and sex-matched controls (2-tailed). There are limited fundamental movement and muscle activity data for identifying different mechanisms for SI in children and adolescents that can inform subgrouping and treatment allocation. METHODS: Young people between 8 and 18 years were recruited into 2 groups of SI and age- and sex-matched controls (CG). All forms of SI were included, and young people with coexisting neurologic pathologies or deficits were excluded. Participants attended a single session and carried out 4 unweighted and 3 weighted tasks in which their movements and muscle activity was measured using 3-dimensional (3D) movement analysis and surface electromyography (sEMG). Statistical parametric mapping was used to identify between-group differences. RESULTS: Data were collected for 30 young people (15 SI [6 male, 9 female] and 15 CG [8 male, 7 female]). The mean (standard deviation) age of the participants was 13.6 years (3.0). The SI group demonstrated consistently more protracted and elevated sternoclavicular joint positions during all movements. Normalized muscle activity in latissimus dorsi was lower in the SI group and had the most statistically significant differences across all movements. Where differences were identified, the SI group also had increased normalized activity of their middle trapezius, posterior deltoid, and biceps muscles but decreased activity of their latissimus dorsi, triceps and anterior deltoid muscles compared with the CG group. No statistically significant differences were found for the pectoralis major across any movements. Weighted tasks produced fewer differences in muscle activity patterns compared with unweighted tasks. DISCUSSION AND CONCLUSION: Young people with SI may adapt their movements to minimize glenohumeral joint instability. This was demonstrated by reduced variability in acromioclavicular and sternoclavicular joint angles, adoption of different movement strategies across the same joints, and increased activity of the scapular stabilizing muscles, despite achieving similar arm positions to the CG. Young people with SI demonstrated consistent differences in their muscle activity and movement patterns. Consistently observed differences at the shoulder girdle included increased sternoclavicular protraction and elevation accompanied by increased normalized activity of the posterior scapula-stabilizing muscles. Existing methods of measurement may be used to inform clinical decision making; however, further work is needed to evaluate the prognostic and clinical utility of derived 3D and sEMG data for informing decision making within SI.

Quantification of long-term nonlinear stress relaxation of bovine trabecular bone.

The reliability of computational models in orthopedic biomechanics depends often on the accuracy of the bone material properties. It is widely recognized that the mechanical response of trabecular bone is time-dependent, yet it is often ignored for the sake of simplicity. Previous investigations into the viscoelastic properties of trabecular bone have not explored the relationship between nonlinear stress relaxation and bone mineral density. The inclusion of this behavior could enhance the accuracy of simulations of orthopedic interventions, such as of primary fixation of implants. Although methods to quantify the viscoelastic behavior are known, the time period during which the viscoelastic properties should be investigated to obtain reliable predictions is currently unclear. Therefore, this study aimed to: 1) Investigate the duration of stress relaxation in bovine trabecular bone; 2) construct a material model that describes the nonlinear viscoelastic behavior of uniaxial stress relaxation experiments on trabecular bone; and 3) implement bone density into this model. Uniaxial compressive stress relaxation experiments were performed with cylindrical bovine femoral trabecular bone samples (n = 16) with constant strain held for 24 h. Additionally, multiple stress relaxation experiments with four ascending strain levels with a holding time of 30 min, based on the results of the 24-h experiment, were executed on 18 bovine bone cores. The bone specimens used in this study had a mean diameter of 12.80 mm and a mean height of 28.70 mm. A Schapery and a Superposition model were used to capture the nonlinear stress relaxation behavior in terms of applied strain level and bone mineral density. While most stress relaxation happened in the first 10 min (up to 53 %) after initial compression, the stress relaxation continued even after 24 h. Up to 69 % of stress relaxation was observed at 24 h. Extrapolating the results of 30 min of experimental data to 24 h provided a good fit for accuracy with much improved experimental efficiency. The Schapery and Superposition model were both capable of fitting the repeated stress relaxation in a sample-by-sample approach. However, since bone mineral density did not influence the time-dependent behavior, only the Superposition model could be used for a group-based model fit. Although the sample-by-sample approach was more accurate for an individual specimen, the group based approach is considered a useful model for general application.

Local large language models for privacy-preserving accelerated review of historic echocardiogram reports.

OBJECTIVES: The study developed framework that leverages an open-source Large Language Model (LLM) to enable clinicians to ask plain-language questions about a patient's entire echocardiogram report history. This approach is intended to streamline the extraction of clinical insights from multiple echocardiogram reports, particularly in patients with complex cardiac diseases, thereby enhancing both patient care and research efficiency. MATERIALS AND METHODS: Data from over 10 years were collected, comprising echocardiogram reports from patients with more than 10 echocardiograms on file at the Mount Sinai Health System. These reports were converted into a single document per patient for analysis, broken down into snippets and relevant snippets were retrieved using text similarity measures. The LLaMA-2 70B model was employed for analyzing the text using a specially crafted prompt. The model's performance was evaluated against ground-truth answers created by faculty cardiologists. RESULTS: The study analyzed 432 reports from 37 patients for a total of 100 question-answer pairs. The LLM correctly answered 90% questions, with accuracies of 83% for temporality, 93% for severity assessment, 84% for intervention identification, and 100% for diagnosis retrieval. Errors mainly stemmed from the LLM's inherent limitations, such as misinterpreting numbers or hallucinations. CONCLUSION: The study demonstrates the feasibility and effectiveness of using a local, open-source LLM for querying and interpreting echocardiogram report data. This approach offers a significant improvement over traditional keyword-based searches, enabling more contextually relevant and semantically accurate responses; in turn showing promise in enhancing clinical decision-making and research by facilitating more efficient access to complex patient data.

Assessing calibration and bias of a deployed machine learning malnutrition prediction model within a large healthcare system.

Malnutrition is a frequently underdiagnosed condition leading to increased morbidity, mortality, and healthcare costs. The Mount Sinai Health System (MSHS) deployed a machine learning model (MUST-Plus) to detect malnutrition upon hospital admission. However, in diverse patient groups, a poorly calibrated model may lead to misdiagnosis, exacerbating health care disparities. We explored the model's calibration across different variables and methods to improve calibration. Data from adult patients admitted to five MSHS hospitals from January 1, 2021 - December 31, 2022, were analyzed. We compared MUST-Plus prediction to the registered dietitian's formal assessment. Hierarchical calibration was assessed and compared between the recalibration sample (N = 49,562) of patients admitted between January 1, 2021 - December 31, 2022, and the hold-out sample (N = 17,278) of patients admitted between January 1, 2023 - September 30, 2023. Statistical differences in calibration metrics were tested using bootstrapping with replacement. Before recalibration, the overall model calibration intercept was -1.17 (95% CI: -1.20, -1.14), slope was 1.37 (95% CI: 1.34, 1.40), and Brier score was 0.26 (95% CI: 0.25, 0.26). Both weak and moderate measures of calibration were significantly different between White and Black patients and between male and female patients. Logistic recalibration significantly improved calibration of the model across race and gender in the hold-out sample. The original MUST-Plus model showed significant differences in calibration between White vs. Black patients. It also overestimated malnutrition in females compared to males. Logistic recalibration effectively reduced miscalibration across all patient subgroups. Continual monitoring and timely recalibration can improve model accuracy.

A National Institutes of Health Approach for Advancing Research to Improve Youth Mental Health and Reduce Disparities.

Even before the COVID-19 pandemic, mental health challenges were the leading cause of disability and poor health outcomes in youth. Challenges are even greater for youth from racially and ethnically minoritized groups in the United States. Racially and ethnically minoritized youth are more vulnerable to mental health problems than White adolescents, yet are less likely to use mental health services. In late 2021, the National Institutes of Health (NIH) sponsored a virtual conference to examine the state of the science around youth mental health disparities (YMHD), focusing on youth from racially and ethnically minoritized populations and the intersection of race and ethnicity with other drivers of mental health disparities. Key findings and feedback gleaned from the conference have informed strategic planning processes related to YMHD, which has included the development of a strategic framework and funding opportunities, designed to reduce YMHD. This commentary briefly describes the collaborative approach used to develop this framework and other strategies implemented across the NIH to address YMHD and serves as an urgent call to action.

Evaluating the accuracy of a state-of-the-art large language model for prediction of admissions from the emergency room.

BACKGROUND: Artificial intelligence (AI) and large language models (LLMs) can play a critical role in emergency room operations by augmenting decision-making about patient admission. However, there are no studies for LLMs using real-world data and scenarios, in comparison to and being informed by traditional supervised machine learning (ML) models. We evaluated the performance of GPT-4 for predicting patient admissions from emergency department (ED) visits. We compared performance to traditional ML models both naively and when informed by few-shot examples and/or numerical probabilities. METHODS: We conducted a retrospective study using electronic health records across 7 NYC hospitals. We trained Bio-Clinical-BERT and XGBoost (XGB) models on unstructured and structured data, respectively, and created an ensemble model reflecting ML performance. We then assessed GPT-4 capabilities in many scenarios: through Zero-shot, Few-shot with and without retrieval-augmented generation (RAG), and with and without ML numerical probabilities. RESULTS: The Ensemble ML model achieved an area under the receiver operating characteristic curve (AUC) of 0.88, an area under the precision-recall curve (AUPRC) of 0.72 and an accuracy of 82.9%. The naïve GPT-4's performance (0.79 AUC, 0.48 AUPRC, and 77.5% accuracy) showed substantial improvement when given limited, relevant data to learn from (ie, RAG) and underlying ML probabilities (0.87 AUC, 0.71 AUPRC, and 83.1% accuracy). Interestingly, RAG alone boosted performance to near peak levels (0.82 AUC, 0.56 AUPRC, and 81.3% accuracy). CONCLUSIONS: The naïve LLM had limited performance but showed significant improvement in predicting ED admissions when supplemented with real-world examples to learn from, particularly through RAG, and/or numerical probabilities from traditional ML models. Its peak performance, although slightly lower than the pure ML model, is noteworthy given its potential for providing reasoning behind predictions. Further refinement of LLMs with real-world data is necessary for successful integration as decision-support tools in care settings.

Development and Validation of a Deep Learning Classifier Using Chest Radiographs to Predict Extubation Success in Patients Undergoing Invasive Mechanical Ventilation.

The decision to extubate patients on invasive mechanical ventilation is critical; however, clinician performance in identifying patients to liberate from the ventilator is poor. Machine Learning-based predictors using tabular data have been developed; however, these fail to capture the wide spectrum of data available. Here, we develop and validate a deep learning-based model using routinely collected chest X-rays to predict the outcome of attempted extubation. We included 2288 serial patients admitted to the Medical ICU at an urban academic medical center, who underwent invasive mechanical ventilation, with at least one intubated CXR, and a documented extubation attempt. The last CXR before extubation for each patient was taken and split 79/21 for training/testing sets, then transfer learning with k-fold cross-validation was used on a pre-trained ResNet50 deep learning architecture. The top three models were ensembled to form a final classifier. The Grad-CAM technique was used to visualize image regions driving predictions. The model achieved an AUC of 0.66, AUPRC of 0.94, sensitivity of 0.62, and specificity of 0.60. The model performance was improved compared to the Rapid Shallow Breathing Index (AUC 0.61) and the only identified previous study in this domain (AUC 0.55), but significant room for improvement and experimentation remains.

Evaluating prompt engineering on GPT-3.5's performance in USMLE-style medical calculations and clinical scenarios generated by GPT-4.

This study was designed to assess how different prompt engineering techniques, specifically direct prompts, Chain of Thought (CoT), and a modified CoT approach, influence the ability of GPT-3.5 to answer clinical and calculation-based medical questions, particularly those styled like the USMLE Step 1 exams. To achieve this, we analyzed the responses of GPT-3.5 to two distinct sets of questions: a batch of 1000 questions generated by GPT-4, and another set comprising 95 real USMLE Step 1 questions. These questions spanned a range of medical calculations and clinical scenarios across various fields and difficulty levels. Our analysis revealed that there were no significant differences in the accuracy of GPT-3.5's responses when using direct prompts, CoT, or modified CoT methods. For instance, in the USMLE sample, the success rates were 61.7% for direct prompts, 62.8% for CoT, and 57.4% for modified CoT, with a p-value of 0.734. Similar trends were observed in the responses to GPT-4 generated questions, both clinical and calculation-based, with p-values above 0.05 indicating no significant difference between the prompt types. The conclusion drawn from this study is that the use of CoT prompt engineering does not significantly alter GPT-3.5's effectiveness in handling medical calculations or clinical scenario questions styled like those in USMLE exams. This finding is crucial as it suggests that performance of ChatGPT remains consistent regardless of whether a CoT technique is used instead of direct prompts. This consistency could be instrumental in simplifying the integration of AI tools like ChatGPT into medical education, enabling healthcare professionals to utilize these tools with ease, without the necessity for complex prompt engineering.

Derivation, External Validation and Clinical Implications of a deep learning approach for intracranial pressure estimation using non-cranial waveform measurements.

Importance: Increased intracranial pressure (ICP) is associated with adverse neurological outcomes, but needs invasive monitoring. Objective: Development and validation of an AI approach for detecting increased ICP (aICP) using only non-invasive extracranial physiological waveform data. Design: Retrospective diagnostic study of AI-assisted detection of increased ICP. We developed an AI model using exclusively extracranial waveforms, externally validated it and assessed associations with clinical outcomes. Setting: MIMIC-III Waveform Database (2000-2013), a database derived from patients admitted to an ICU in an academic Boston hospital, was used for development of the aICP model, and to report association with neurologic outcomes. Data from Mount Sinai Hospital (2020-2022) in New York City was used for external validation. Participants: Patients were included if they were older than 18 years, and were monitored with electrocardiograms, arterial blood pressure, respiratory impedance plethysmography and pulse oximetry. Patients who additionally had intracranial pressure monitoring were used for development (N=157) and external validation (N=56). Patients without intracranial monitors were used for association with outcomes (N=1694). Exposures: Extracranial waveforms including electrocardiogram, arterial blood pressure, plethysmography and SpO2. Main Outcomes and Measures: Intracranial pressure > 15 mmHg. Measures were Area under receiver operating characteristic curves (AUROCs), sensitivity, specificity, and accuracy at threshold of 0.5. We calculated odds ratios and p-values for phenotype association. Results: The AUROC was 0.91 (95% CI, 0.90-0.91) on testing and 0.80 (95% CI, 0.80-0.80) on external validation. aICP had accuracy, sensitivity, and specificity of 73.8% (95% CI, 72.0%-75.6%), 99.5% (95% CI 99.3%-99.6%), and 76.9% (95% CI, 74.0-79.8%) on external validation. A ten-percentile increment was associated with stroke (OR=2.12; 95% CI, 1.27-3.13), brain malignancy (OR=1.68; 95% CI, 1.09-2.60), subdural hemorrhage (OR=1.66; 95% CI, 1.07-2.57), intracerebral hemorrhage (OR=1.18; 95% CI, 1.07-1.32), and procedures like percutaneous brain biopsy (OR=1.58; 95% CI, 1.15-2.18) and craniotomy (OR = 1.43; 95% CI, 1.12-1.84; P < 0.05 for all). Conclusions and Relevance: aICP provides accurate, non-invasive estimation of increased ICP, and is associated with neurological outcomes and neurosurgical procedures in patients without intracranial monitoring.