Merlin: A Vision Language Foundation Model for 3D Computed Tomography.

Over 85 million computed tomography (CT) scans are performed annually in the US, of which approximately one quarter focus on the abdomen. Given the current shortage of both general and specialized radiologists, there is a large impetus to use artificial intelligence to alleviate the burden of interpreting these complex imaging studies while simultaneously using the images to extract novel physiological insights. Prior state-of-the-art approaches for automated medical image interpretation leverage vision language models (VLMs) that utilize both the image and the corresponding textual radiology reports. However, current medical VLMs are generally limited to 2D images and short reports. To overcome these shortcomings for abdominal CT interpretation, we introduce Merlin - a 3D VLM that leverages both structured electronic health records (EHR) and unstructured radiology reports for pretraining without requiring additional manual annotations. We train Merlin using a high-quality clinical dataset of paired CT scans (6+ million images from 15,331 CTs), EHR diagnosis codes (1.8+ million codes), and radiology reports (6+ million tokens) for training. We comprehensively evaluate Merlin on 6 task types and 752 individual tasks. The non-adapted (off-the-shelf) tasks include zero-shot findings classification (31 findings), phenotype classification (692 phenotypes), and zero-shot cross-modal retrieval (image to findings and image to impressions), while model adapted tasks include 5-year chronic disease prediction (6 diseases), radiology report generation, and 3D semantic segmentation (20 organs). We perform internal validation on a test set of 5,137 CTs, and external validation on 7,000 clinical CTs and on two public CT datasets (VerSe, TotalSegmentator). Beyond these clinically-relevant evaluations, we assess the efficacy of various network architectures and training strategies to depict that Merlin has favorable performance to existing task-specific baselines. We derive data scaling laws to empirically assess training data needs for requisite downstream task performance. Furthermore, unlike conventional VLMs that require hundreds of GPUs for training, we perform all training on a single GPU. This computationally efficient design can help democratize foundation model training, especially for health systems with compute constraints. We plan to release our trained models, code, and dataset, pending manual removal of all protected health information.

ShapeMed-Knee: A Dataset and Neural Shape Model Benchmark for Modeling 3D Femurs.

Analyzing anatomic shapes of tissues and organs is pivotal for accurate disease diagnostics and clinical decision-making. One prominent disease that depends on anatomic shape analysis is osteoarthritis, which affects 30 million Americans. To advance osteoarthritis diagnostics and prognostics, we introduce ShapeMed-Knee, a 3D shape dataset with 9,376 high-resolution, medical-imaging-based 3D shapes of both femur bone and cartilage. Besides data, ShapeMed-Knee includes two benchmarks for assessing reconstruction accuracy and five clinical prediction tasks that assess the utility of learned shape representations. Leveraging ShapeMed-Knee, we develop and evaluate a novel hybrid explicit-implicit neural shape model which achieves up to 40% better reconstruction accuracy than a statistical shape model and implicit neural shape model. Our hybrid models achieve state-of-the-art performance for preserving cartilage biomarkers; they're also the first models to successfully predict localized structural features of osteoarthritis, outperforming shape models and convolutional neural networks applied to raw magnetic resonance images and segmentations. The ShapeMed-Knee dataset provides medical evaluations to reconstruct multiple anatomic surfaces and embed meaningful disease-specific information. ShapeMed-Knee reduces barriers to applying 3D modeling in medicine, and our benchmarks highlight that advancements in 3D modeling can enhance the diagnosis and risk stratification for complex diseases. The dataset, code, and benchmarks will be made freely accessible.

Automated abdominal CT contrast phase detection using an interpretable and open-source artificial intelligence algorithm.

OBJECTIVES: To develop and validate an open-source artificial intelligence (AI) algorithm to accurately detect contrast phases in abdominal CT scans. MATERIALS AND METHODS: Retrospective study aimed to develop an AI algorithm trained on 739 abdominal CT exams from 2016 to 2021, from 200 unique patients, covering 1545 axial series. We performed segmentation of five key anatomic structures-aorta, portal vein, inferior vena cava, renal parenchyma, and renal pelvis-using TotalSegmentator, a deep learning-based tool for multi-organ segmentation, and a rule-based approach to extract the renal pelvis. Radiomics features were extracted from the anatomical structures for use in a gradient-boosting classifier to identify four contrast phases: non-contrast, arterial, venous, and delayed. Internal and external validation was performed using the F1 score and other classification metrics, on the external dataset "VinDr-Multiphase CT". RESULTS: The training dataset consisted of 172 patients (mean age, 70 years ± 8, 22% women), and the internal test set included 28 patients (mean age, 68 years ± 8, 14% women). In internal validation, the classifier achieved an accuracy of 92.3%, with an average F1 score of 90.7%. During external validation, the algorithm maintained an accuracy of 90.1%, with an average F1 score of 82.6%. Shapley feature attribution analysis indicated that renal and vascular radiodensity values were the most important for phase classification. CONCLUSION: An open-source and interpretable AI algorithm accurately detects contrast phases in abdominal CT scans, with high accuracy and F1 scores in internal and external validation, confirming its generalization capability. CLINICAL RELEVANCE STATEMENT: Contrast phase detection in abdominal CT scans is a critical step for downstream AI applications, deploying algorithms in the clinical setting, and for quantifying imaging biomarkers, ultimately allowing for better diagnostics and increased access to diagnostic imaging. KEY POINTS: Digital Imaging and Communications in Medicine labels are inaccurate for determining the abdominal CT scan phase. AI provides great help in accurately discriminating the contrast phase. Accurate contrast phase determination aids downstream AI applications and biomarker quantification.

Abdominal CT metrics in 17,646 patients reveal associations between myopenia, myosteatosis, and medical phenotypes: a phenome-wide association study.

BACKGROUND: Deep learning facilitates large-scale automated imaging evaluation of body composition. However, associations of body composition biomarkers with medical phenotypes have been underexplored. Phenome-wide association study (PheWAS) techniques search for medical phenotypes associated with biomarkers. A PheWAS integrating large-scale analysis of imaging biomarkers and electronic health record (EHR) data could discover previously unreported associations and validate expected associations. Here we use PheWAS methodology to determine the association of abdominal CT-based skeletal muscle metrics with medical phenotypes in a large North American cohort. METHODS: An automated deep learning pipeline was used to measure skeletal muscle index (SMI; biomarker of myopenia) and skeletal muscle density (SMD; biomarker of myosteatosis) from abdominal CT scans of adults between 2012 and 2018. A PheWAS was performed with logistic regression using patient sex and age as covariates to assess for associations between CT-derived muscle metrics and 611 common EHR-derived medical phenotypes. PheWAS P values were considered significant at a Bonferroni corrected threshold (α = 0.05/1222). FINDINGS: 17,646 adults (mean age, 56 years ± 19 [SD]; 57.5% women) were included. CT-derived SMI was significantly associated with 268 medical phenotypes; SMD with 340 medical phenotypes. Previously unreported associations with the highest magnitude of significance included higher SMI with decreased cardiac dysrhythmias (OR [95% CI], 0.59 [0.55-0.64]; P < 0.0001), decreased epilepsy (OR, 0.59 [0.50-0.70]; P < 0.0001), and increased elevated prostate-specific antigen (OR, 1.84 [1.47-2.31]; P < 0.0001), and higher SMD with decreased decubitus ulcers (OR, 0.36 [0.31-0.42]; P < 0.0001), sleep disorders (OR, 0.39 [0.32-0.47]; P < 0.0001), and osteomyelitis (OR, 0.43 [0.36-0.52]; P < 0.0001). INTERPRETATION: PheWAS methodology reveals previously unreported associations between CT-derived biomarkers of myopenia and myosteatosis and EHR medical phenotypes. The high-throughput PheWAS technique applied on a population scale can generate research hypotheses related to myopenia and myosteatosis and can be adapted to research possible associations of other imaging biomarkers with hundreds of EHR medical phenotypes. FUNDING: National Institutes of Health, Stanford AIMI-HAI pilot grant, Stanford Precision Health and Integrated Diagnostics, Stanford Cardiovascular Institute, Stanford Center for Digital Health, and Stanford Knight-Hennessy Scholars.

Adapted large language models can outperform medical experts in clinical text summarization.

Analyzing vast textual data and summarizing key information from electronic health records imposes a substantial burden on how clinicians allocate their time. Although large language models (LLMs) have shown promise in natural language processing (NLP) tasks, their effectiveness on a diverse range of clinical summarization tasks remains unproven. Here we applied adaptation methods to eight LLMs, spanning four distinct clinical summarization tasks: radiology reports, patient questions, progress notes and doctor-patient dialogue. Quantitative assessments with syntactic, semantic and conceptual NLP metrics reveal trade-offs between models and adaptation methods. A clinical reader study with 10 physicians evaluated summary completeness, correctness and conciseness; in most cases, summaries from our best-adapted LLMs were deemed either equivalent (45%) or superior (36%) compared with summaries from medical experts. The ensuing safety analysis highlights challenges faced by both LLMs and medical experts, as we connect errors to potential medical harm and categorize types of fabricated information. Our research provides evidence of LLMs outperforming medical experts in clinical text summarization across multiple tasks. This suggests that integrating LLMs into clinical workflows could alleviate documentation burden, allowing clinicians to focus more on patient care.

Biomarkers of Body Composition.

The importance and impact of imaging biomarkers has been increasing over the past few decades. We review the relevant clinical and imaging terminology needed to understand the clinical and research applications of body composition. Imaging biomarkers of bone, muscle, and fat tissues obtained with dual-energy X-ray absorptiometry, computed tomography, magnetic resonance imaging, and ultrasonography are described.

Skeletal Muscle Area on CT: Determination of an Optimal Height Scaling Power and Testing for Mortality Risk Prediction.

BACKGROUND. Sarcopenia is commonly assessed on CT by use of the skeletal muscle index (SMI), which is calculated as the skeletal muscle area (SMA) at L3 divided by patient height squared (i.e., a height scaling power of 2). OBJECTIVE. The purpose of this study was to determine the optimal height scaling power for SMA measurements on CT and to test the influence of the derived optimal scaling power on the utility of SMI in predicting all-cause mortality. METHODS. This retrospective study included 16,575 patients (6985 men, 9590 women; mean age, 56.4 years) who underwent abdominal CT from December 2012 through October 2018. The SMA at L3 was determined using automated software. The sample was stratified into two groups: 5459 patients without major medical conditions (based on ICD-9 and ICD-10 codes) who were included in the analysis for determining the optimal height scaling power and 11,116 patients with major medical conditions who were included for the purpose of testing this power. The optimal scaling power was determined by allometric analysis (whereby regression coefficients were fitted to log-linear sex-specific models relating height to SMA) and by analysis of statistical independence of SMI from height across scaling powers. Cox proportional hazards models were used to test the influence of the derived optimal scaling power on the utility of SMI in predicting all-cause mortality. RESULTS. In allometric analysis, the regression coefficient of log(height) in patients 40 years old and younger was 1.02 in men and 1.08 in women, and in patients older than 40 years old, it was 1.07 in men and 1.10 in women (all p < .05 vs regression coefficient of 2). In analyses for statistical independence of SMI from height, the optimal height scaling power (i.e., those yielding correlations closest to 0) was, in patients 40 years old and younger, 0.97 in men and 1.08 in women, whereas in patients older than 40 years old, it was 1.03 in men and 1.09 in women. In the Cox model used for testing, SMI predicted all-cause mortality with a higher concordance index using of a height scaling power of 1 rather than 2 in men (0.675 vs 0.663, p < .001) and in women (0.664 vs 0.653, p < .001). CONCLUSION. The findings support a height scaling power of 1, rather than a conventional power of 2, for SMI computation. CLINICAL IMPACT. A revised height scaling power for SMI could impact the utility of CT-based sarcopenia diagnoses in risk assessment.

Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts.

Sifting through vast textual data and summarizing key information from electronic health records (EHR) imposes a substantial burden on how clinicians allocate their time. Although large language models (LLMs) have shown immense promise in natural language processing (NLP) tasks, their efficacy on a diverse range of clinical summarization tasks has not yet been rigorously demonstrated. In this work, we apply domain adaptation methods to eight LLMs, spanning six datasets and four distinct clinical summarization tasks: radiology reports, patient questions, progress notes, and doctor-patient dialogue. Our thorough quantitative assessment reveals trade-offs between models and adaptation methods in addition to instances where recent advances in LLMs may not improve results. Further, in a clinical reader study with ten physicians, we show that summaries from our best-adapted LLMs are preferable to human summaries in terms of completeness and correctness. Our ensuing qualitative analysis highlights challenges faced by both LLMs and human experts. Lastly, we correlate traditional quantitative NLP metrics with reader study scores to enhance our understanding of how these metrics align with physician preferences. Our research marks the first evidence of LLMs outperforming human experts in clinical text summarization across multiple tasks. This implies that integrating LLMs into clinical workflows could alleviate documentation burden, empowering clinicians to focus more on personalized patient care and the inherently human aspects of medicine.

Combinatorial Polyacrylamide Hydrogels for Preventing Biofouling on Implantable Biosensors.

Biofouling on the surface of implanted medical devices and biosensors severely hinders device functionality and drastically shortens device lifetime. Poly(ethylene glycol) and zwitterionic polymers are currently considered "gold-standard" device coatings to reduce biofouling. To discover novel anti-biofouling materials, a combinatorial library of polyacrylamide-based copolymer hydrogels is created, and their ability is screened to prevent fouling from serum and platelet-rich plasma in a high-throughput parallel assay. It is found that certain nonintuitive copolymer compositions exhibit superior anti-biofouling properties over current gold-standard materials, and machine learning is used to identify key molecular features underpinning their performance. For validation, the surfaces of electrochemical biosensors are coated with hydrogels and their anti-biofouling performance in vitro and in vivo in rodent models is evaluated. The copolymer hydrogels preserve device function and enable continuous measurements of a small-molecule drug in vivo better than gold-standard coatings. The novel methodology described enables the discovery of anti-biofouling materials that can extend the lifetime of real-time in vivo sensing devices.

Confined Liquid-Phase Growth of Crystalline Compound Semiconductors on Any Substrate.

The growth of crystalline compound semiconductors on amorphous and non-epitaxial substrates is a fundamental challenge for state-of-the-art thin-film epitaxial growth techniques. Direct growth of materials on technologically relevant amorphous surfaces, such as nitrides or oxides results in nanocrystalline thin films or nanowire-type structures, preventing growth and integration of high-performance devices and circuits on these surfaces. Here, we show crystalline compound semiconductors grown directly on technologically relevant amorphous and non-epitaxial substrates in geometries compatible with standard microfabrication technology. Furthermore, by removing the traditional epitaxial constraint, we demonstrate an atomically sharp lateral heterojunction between indium phosphide and tin phosphide, two materials with vastly different crystal structures, a structure that cannot be grown with standard vapor-phase growth approaches. Critically, this approach enables the growth and manufacturing of crystalline materials without requiring a nearly lattice-matched substrate, potentially impacting a wide range of fields, including electronics, photonics, and energy devices.

Scalable Indium Phosphide Thin-Film Nanophotonics Platform for Photovoltaic and Photoelectrochemical Devices.

Recent developments in nanophotonics have provided a clear roadmap for improving the efficiency of photonic devices through control over absorption and emission of devices. These advances could prove transformative for a wide variety of devices, such as photovoltaics, photoelectrochemical devices, photodetectors, and light-emitting diodes. However, it is often challenging to physically create the nanophotonic designs required to engineer the optical properties of devices. Here, we present a platform based on crystalline indium phosphide that enables thin-film nanophotonic structures with physical morphologies that are impossible to achieve through conventional state-of-the-art material growth techniques. Here, nanostructured InP thin films have been demonstrated on non-epitaxial alumina inverted nanocone (i-cone) substrates via a low-cost and scalable thin-film vapor-liquid-solid growth technique. In this process, indium films are first evaporated onto the i-cone structures in the desired morphology, followed by a high-temperature step that causes a phase transformation of the indium into indium phosphide, preserving the original morphology of the deposited indium. Through this approach, a wide variety of nanostructured film morphologies are accessible using only control over evaporation process variables. Critically, the as-grown nanotextured InP thin films demonstrate excellent optoelectronic properties, suggesting this platform is promising for future high-performance nanophotonic devices.

Bandgap Control via Structural and Chemical Tuning of Transition Metal Perovskite Chalcogenides.

Transition metal perovskite chalcogenides are a new class of versatile semiconductors with high absorption coefficient and luminescence efficiency. Polycrystalline materials synthesized by an iodine-catalyzed solid-state reaction show distinctive optical colors and tunable bandgaps across the visible range in photoluminescence, with one of the materials' external efficiency approaching the level of single-crystal InP and CdSe.