Merlin: A Vision Language Foundation Model for 3D Computed Tomography.

Over 85 million computed tomography (CT) scans are performed annually in the US, of which approximately one quarter focus on the abdomen. Given the current shortage of both general and specialized radiologists, there is a large impetus to use artificial intelligence to alleviate the burden of interpreting these complex imaging studies while simultaneously using the images to extract novel physiological insights. Prior state-of-the-art approaches for automated medical image interpretation leverage vision language models (VLMs) that utilize both the image and the corresponding textual radiology reports. However, current medical VLMs are generally limited to 2D images and short reports. To overcome these shortcomings for abdominal CT interpretation, we introduce Merlin - a 3D VLM that leverages both structured electronic health records (EHR) and unstructured radiology reports for pretraining without requiring additional manual annotations. We train Merlin using a high-quality clinical dataset of paired CT scans (6+ million images from 15,331 CTs), EHR diagnosis codes (1.8+ million codes), and radiology reports (6+ million tokens) for training. We comprehensively evaluate Merlin on 6 task types and 752 individual tasks. The non-adapted (off-the-shelf) tasks include zero-shot findings classification (31 findings), phenotype classification (692 phenotypes), and zero-shot cross-modal retrieval (image to findings and image to impressions), while model adapted tasks include 5-year chronic disease prediction (6 diseases), radiology report generation, and 3D semantic segmentation (20 organs). We perform internal validation on a test set of 5,137 CTs, and external validation on 7,000 clinical CTs and on two public CT datasets (VerSe, TotalSegmentator). Beyond these clinically-relevant evaluations, we assess the efficacy of various network architectures and training strategies to depict that Merlin has favorable performance to existing task-specific baselines. We derive data scaling laws to empirically assess training data needs for requisite downstream task performance. Furthermore, unlike conventional VLMs that require hundreds of GPUs for training, we perform all training on a single GPU. This computationally efficient design can help democratize foundation model training, especially for health systems with compute constraints. We plan to release our trained models, code, and dataset, pending manual removal of all protected health information.

Diagnostic Accuracy of Quantitative Multicontrast 5-Minute Knee MRI Using Prospective Artificial Intelligence Image Quality Enhancement.

BACKGROUND. Potential approaches for abbreviated knee MRI, including prospective acceleration with deep learning, have achieved limited clinical implementation. OBJECTIVE. The objective of this study was to evaluate the interreader agreement between conventional knee MRI and a 5-minute 3D quantitative double-echo steady-state (qDESS) sequence with automatic T2 mapping and deep learning super-resolutionaugmentation and to compare the diagnostic performance of the two methods regarding findings from arthroscopic surgery. METHODS. Fifty-one patients with knee pain underwent knee MRI that included an additional 3D qDESS sequence with automatic T2 mapping. Fourier interpolation was followed by prospective deep learning super resolution to enhance qDESS slice resolution twofold. A musculoskeletal radiologist and a radiology resident performed independent retrospective evaluations of articular cartilage, menisci, ligaments, bones, extensor mechanism, and synovium using conventional MRI. Following a 2-month washout period, readers reviewed qDESS images alone followed by qDESS with the automatic T2 maps. Interreader agreement between conventional MRI and qDESS was computed using percentage agreement and Cohen kappa. The sensitivity and specificity of conventional MRI, qDESS alone, and qDESS plus T2 mapping were compared with arthroscopic findings using exact McNemar tests. RESULTS. Conventional MRI and qDESS showed 92% agreement in evaluating all tissues. Kappa was 0.79 (95% CI, 0.76-0.81) across all imaging findings. In 43 patients who underwent arthroscopy, sensitivity and specificity were not significantly different (p = .23 to > .99) between conventional MRI (sensitivity, 58-93%; specificity, 27-87%) and qDESS alone (sensitivity, 54-90%; specificity, 23-91%) for cartilage, menisci, ligaments, and synovium. For grade 1 cartilage lesions, sensitivity and specificity were 33% and 56%, respectively, for conventional MRI; 23% and 53% for qDESS (p = .81); and 46% and 39% for qDESS with T2 mapping (p = .80). For grade 2A lesions, values were 27% and 53% for conventional MRI, 26% and 52% for qDESS (p = .02), and 58% and 40% for qDESS with T2 mapping (p < .001). CONCLUSION. The qDESS method prospectively augmented with deep learning showed strong interreader agreement with conventional knee MRI and near-equivalent diagnostic performance regarding arthroscopy. The ability of qDESS to automatically generate T2 maps increases sensitivity for cartilage abnormalities. CLINICAL IMPACT. Using prospective artificial intelligence to enhance qDESS image quality may facilitate an abbreviated knee MRI protocol while generating quantitative T2 maps.

Utility of deep learning super-resolution in the context of osteoarthritis MRI biomarkers.

BACKGROUND: Super-resolution is an emerging method for enhancing MRI resolution; however, its impact on image quality is still unknown. PURPOSE: To evaluate MRI super-resolution using quantitative and qualitative metrics of cartilage morphometry, osteophyte detection, and global image blurring. STUDY TYPE: Retrospective. POPULATION: In all, 176 MRI studies of subjects at varying stages of osteoarthritis. FIELD STRENGTH/SEQUENCE: Original-resolution 3D double-echo steady-state (DESS) and DESS with 3× thicker slices retrospectively enhanced using super-resolution and tricubic interpolation (TCI) at 3T. ASSESSMENT: A quantitative comparison of femoral cartilage morphometry was performed for the original-resolution DESS, the super-resolution, and the TCI scans in 17 subjects. A reader study by three musculoskeletal radiologists assessed cartilage image quality, overall image sharpness, and osteophytes incidence in all three sets of scans. A referenceless blurring metric evaluated blurring in all three image dimensions for the three sets of scans. STATISTICAL TESTS: Mann-Whitney U-tests compared Dice coefficients (DC) of segmentation accuracy for the DESS, super-resolution, and TCI images, along with the image quality readings and blurring metrics. Sensitivity, specificity, and diagnostic odds ratio (DOR) with 95% confidence intervals compared osteophyte detection for the super-resolution and TCI images, with the original-resolution as a reference. RESULTS: DC for the original-resolution (90.2 ± 1.7%) and super-resolution (89.6 ± 2.0%) were significantly higher (P < 0.001) than TCI (86.3 ± 5.6%). Segmentation overlap of super-resolution with the original-resolution (DC = 97.6 ± 0.7%) was significantly higher (P < 0.0001) than TCI overlap (DC = 95.0 ± 1.1%). Cartilage image quality for sharpness and contrast levels, and the through-plane quantitative blur factor for super-resolution images, was significantly (P < 0.001) better than TCI. Super-resolution osteophyte detection sensitivity of 80% (76-82%), specificity of 93% (92-94%), and DOR of 32 (22-46) was significantly higher (P < 0.001) than TCI sensitivity of 73% (69-76%), specificity of 90% (89-91%), and DOR of 17 (13-22). DATA CONCLUSION: Super-resolution appears to consistently outperform naïve interpolation and may improve image quality without biasing quantitative biomarkers. LEVEL OF EVIDENCE: 2 Technical Efficacy: Stage 2 J. Magn. Reson. Imaging 2020;51:768-779.

Super-resolution musculoskeletal MRI using deep learning.

PURPOSE: To develop a super-resolution technique using convolutional neural networks for generating thin-slice knee MR images from thicker input slices, and compare this method with alternative through-plane interpolation methods. METHODS: We implemented a 3D convolutional neural network entitled DeepResolve to learn residual-based transformations between high-resolution thin-slice images and lower-resolution thick-slice images at the same center locations. DeepResolve was trained using 124 double echo in steady-state (DESS) data sets with 0.7-mm slice thickness and tested on 17 patients. Ground-truth images were compared with DeepResolve, clinically used tricubic interpolation, and Fourier interpolation methods, along with state-of-the-art single-image sparse-coding super-resolution. Comparisons were performed using structural similarity, peak SNR, and RMS error image quality metrics for a multitude of thin-slice downsampling factors. Two musculoskeletal radiologists ranked the 3 data sets and reviewed the diagnostic quality of the DeepResolve, tricubic interpolation, and ground-truth images for sharpness, contrast, artifacts, SNR, and overall diagnostic quality. Mann-Whitney U tests evaluated differences among the quantitative image metrics, reader scores, and rankings. Cohen's Kappa (κ) evaluated interreader reliability. RESULTS: DeepResolve had significantly better structural similarity, peak SNR, and RMS error than tricubic interpolation, Fourier interpolation, and sparse-coding super-resolution for all downsampling factors (p < .05, except 4 × and 8 × sparse-coding super-resolution downsampling factors). In the reader study, DeepResolve significantly outperformed (p < .01) tricubic interpolation in all image quality categories and overall image ranking. Both readers had substantial scoring agreement (κ = 0.73). CONCLUSION: DeepResolve was capable of resolving high-resolution thin-slice knee MRI from lower-resolution thicker slices, achieving superior quantitative and qualitative diagnostic performance to both conventionally used and state-of-the-art methods.

Comparison of fMRI analysis methods for heterogeneous BOLD responses in block design studies.

A large number of fMRI studies have shown that the temporal dynamics of evoked BOLD responses can be highly heterogeneous. Failing to model heterogeneous responses in statistical analysis can lead to significant errors in signal detection and characterization and alter the neurobiological interpretation. However, to date it is not clear that, out of a large number of options, which methods are robust against variability in the temporal dynamics of BOLD responses in block-design studies. Here, we used rodent optogenetic fMRI data with heterogeneous BOLD responses and simulations guided by experimental data as a means to investigate different analysis methods' performance against heterogeneous BOLD responses. Evaluations are carried out within the general linear model (GLM) framework and consist of standard basis sets as well as independent component analysis (ICA). Analyses show that, in the presence of heterogeneous BOLD responses, conventionally used GLM with a canonical basis set leads to considerable errors in the detection and characterization of BOLD responses. Our results suggest that the 3rd and 4th order gamma basis sets, the 7th to 9th order finite impulse response (FIR) basis sets, the 5th to 9th order B-spline basis sets, and the 2nd to 5th order Fourier basis sets are optimal for good balance between detection and characterization, while the 1st order Fourier basis set (coherence analysis) used in our earlier studies show good detection capability. ICA has mostly good detection and characterization capabilities, but detects a large volume of spurious activation with the control fMRI data.

Optogenetic Functional MRI.

The investigation of the functional connectivity of precise neural circuits across the entire intact brain can be achieved through optogenetic functional magnetic resonance imaging (ofMRI), which is a novel technique that combines the relatively high spatial resolution of high-field fMRI with the precision of optogenetic stimulation. Fiber optics that enable delivery of specific wavelengths of light deep into the brain in vivo are implanted into regions of interest in order to specifically stimulate targeted cell types that have been genetically induced to express light-sensitive trans-membrane conductance channels, called opsins. fMRI is used to provide a non-invasive method of determining the brain's global dynamic response to optogenetic stimulation of specific neural circuits through measurement of the blood-oxygen-level-dependent (BOLD) signal, which provides an indirect measurement of neuronal activity. This protocol describes the construction of fiber optic implants, the implantation surgeries, the imaging with photostimulation and the data analysis required to successfully perform ofMRI. In summary, the precise stimulation and whole-brain monitoring ability of ofMRI are crucial factors in making ofMRI a powerful tool for the study of the connectomics of the brain in both healthy and diseased states.

Combining optogenetic stimulation and fMRI to validate a multivariate dynamical systems model for estimating causal brain interactions.

State-space multivariate dynamical systems (MDS) (Ryali et al. 2011) and other causal estimation models are being increasingly used to identify directed functional interactions between brain regions. However, the validity and accuracy of such methods are poorly understood. Performance evaluation based on computer simulations of small artificial causal networks can address this problem to some extent, but they often involve simplifying assumptions that reduce biological validity of the resulting data. Here, we use a novel approach taking advantage of recently developed optogenetic fMRI (ofMRI) techniques to selectively stimulate brain regions while simultaneously recording high-resolution whole-brain fMRI data. ofMRI allows for a more direct investigation of causal influences from the stimulated site to brain regions activated downstream and is therefore ideal for evaluating causal estimation methods in vivo. We used ofMRI to investigate whether MDS models for fMRI can accurately estimate causal functional interactions between brain regions. Two cohorts of ofMRI data were acquired, one at Stanford University and the University of California Los Angeles (Cohort 1) and the other at the University of North Carolina Chapel Hill (Cohort 2). In each cohort, optical stimulation was delivered to the right primary motor cortex (M1). General linear model analysis revealed prominent downstream thalamic activation in Cohort 1, and caudate-putamen (CPu) activation in Cohort 2. MDS accurately estimated causal interactions from M1 to thalamus and from M1 to CPu in Cohort 1 and Cohort 2, respectively. As predicted, no causal influences were found in the reverse direction. Additional control analyses demonstrated the specificity of causal interactions between stimulated and target sites. Our findings suggest that MDS state-space models can accurately and reliably estimate causal interactions in ofMRI data and further validate their use for estimating causal interactions in fMRI. More generally, our study demonstrates that the combined use of optogenetics and fMRI provides a powerful new tool for evaluating computational methods designed to estimate causal interactions between distributed brain regions.

High spatial resolution compressed sensing (HSPARSE) functional MRI.

PURPOSE: To propose a novel compressed sensing (CS) high spatial resolution functional MRI (fMRI) method and demonstrate the advantages and limitations of using CS for high spatial resolution fMRI. METHODS: A randomly undersampled variable density spiral trajectory enabling an acceleration factor of 5.3 was designed with a balanced steady state free precession sequence to achieve high spatial resolution data acquisition. A modified k-t SPARSE method was then implemented and applied with a strategy to optimize regularization parameters for consistent, high quality CS reconstruction. RESULTS: The proposed method improves spatial resolution by six-fold with 12 to 47% contrast-to-noise ratio (CNR), 33 to 117% F-value improvement and maintains the same temporal resolution. It also achieves high sensitivity of 69 to 99% compared the original ground-truth, small false positive rate of less than 0.05 and low hemodynamic response function distortion across a wide range of CNRs. The proposed method is robust to physiological noise and enables detection of layer-specific activities in vivo, which cannot be resolved using the highest spatial resolution Nyquist acquisition. CONCLUSION: The proposed method enables high spatial resolution fMRI that can resolve layer-specific brain activity and demonstrates the significant improvement that CS can bring to high spatial resolution fMRI. Magn Reson Med 76:440-455, 2016. © 2015 The Authors. Magnetic Resonance in Medicine published by Wiley Periodicals, Inc. on behalf of International Society for Magnetic Resonance in Medicine. This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.

Frequency-selective control of cortical and subcortical networks by central thalamus.

Central thalamus plays a critical role in forebrain arousal and organized behavior. However, network-level mechanisms that link its activity to brain state remain enigmatic. Here, we combined optogenetics, fMRI, electrophysiology, and video-EEG monitoring to characterize the central thalamus-driven global brain networks responsible for switching brain state. 40 and 100 Hz stimulations of central thalamus caused widespread activation of forebrain, including frontal cortex, sensorimotor cortex, and striatum, and transitioned the brain to a state of arousal in asleep rats. In contrast, 10 Hz stimulation evoked significantly less activation of forebrain, inhibition of sensory cortex, and behavioral arrest. To investigate possible mechanisms underlying the frequency-dependent cortical inhibition, we performed recordings in zona incerta, where 10, but not 40, Hz stimulation evoked spindle-like oscillations. Importantly, suppressing incertal activity during 10 Hz central thalamus stimulation reduced the evoked cortical inhibition. These findings identify key brain-wide dynamics underlying central thalamus arousal regulation.

Optogenetic fMRI reveals distinct, frequency-dependent networks recruited by dorsal and intermediate hippocampus stimulations.

Although the connectivity of hippocampal circuits has been extensively studied, the way in which these connections give rise to large-scale dynamic network activity remains unknown. Here, we used optogenetic fMRI to visualize the brain network dynamics evoked by different frequencies of stimulation of two distinct neuronal populations within dorsal and intermediate hippocampus. Stimulation of excitatory cells in intermediate hippocampus caused widespread cortical and subcortical recruitment at high frequencies, whereas stimulation in dorsal hippocampus led to activity primarily restricted to hippocampus across all frequencies tested. Sustained hippocampal responses evoked during high-frequency stimulation of either location predicted seizure-like afterdischarges in video-EEG experiments, while the widespread activation evoked by high-frequency stimulation of intermediate hippocampus predicted behavioral seizures. A negative BOLD signal observed in dentate gyrus during dorsal, but not intermediate, hippocampus stimulation is proposed to underlie the mechanism for these differences. Collectively, our results provide insight into the dynamic function of hippocampal networks and their role in seizures.

High-throughput optogenetic functional magnetic resonance imaging with parallel computations.

Optogenetic functional magnetic resonance imaging (of MRI) technology enables cell-type-specific, temporally precise neuronal control and the accurate, in vivo readout of the resulting activity across the entire brain. With the ability to precisely control excitation and inhibition parameters and accurately record the resulting activity, there is an increased need for a high-throughput method to bring the of MRI studies to their full potential. In this paper, an advanced system facilitating real-time fMRI with interactive control and analysis in a fraction of the MRI acquisition repetition time (TR) is proposed. With high-processing speed, sufficient time will be available for the integration of future developments that further enhance of MRI data or streamline the study. We designed and implemented a highly optimised, massively parallel system using graphics processing units (GPUs), which achieves the reconstruction, motion correction, and analysis of 3D volume data in approximately 12.80 ms. As a result, with a 750 ms TR and 4 interleaf fMRI acquisition, we can now conduct sliding window reconstruction, motion correction, analysis and display in approximately 1.7% of the TR. Therefore, a significant amount of time can now be allocated to integrating advanced but computationally intensive methods that improve image quality and enhance the analysis results within a TR. Utilising the proposed high-throughput imaging platform with sliding window reconstruction, we were also able to observe the much-debated initial dips in our of MRI data. Combined with methods to further improve SNR, the proposed system will enable efficient real-time, interactive, high-throughput of MRI studies.