Advancing Precision Medicine: Algebraic Topology and Differential Geometry in Radiology and Computational Pathology.

Precision medicine aims to provide personalized care based on individual patient characteristics, rather than guideline-directed therapies for groups of diseases or patient demographics. Images-both radiology- and pathology-derived-are a major source of information on presence, type, and status of disease. Exploring the mathematical relationship of pixels in medical imaging ("radiomics") and cellular-scale structures in digital pathology slides ("pathomics") offers powerful tools for extracting both qualitative and, increasingly, quantitative data. These analytical approaches, however, may be significantly enhanced by applying additional methods arising from fields of mathematics such as differential geometry and algebraic topology that remain underexplored in this context. Geometry's strength lies in its ability to provide precise local measurements, such as curvature, that can be crucial for identifying abnormalities at multiple spatial levels. These measurements can augment the quantitative features extracted in conventional radiomics, leading to more nuanced diagnostics. By contrast, topology serves as a robust shape descriptor, capturing essential features such as connected components and holes. The field of topological data analysis was initially founded to explore the shape of data, with functional network connectivity in the brain being a prominent example. Increasingly, its tools are now being used to explore organizational patterns of physical structures in medical images and digitized pathology slides. By leveraging tools from both differential geometry and algebraic topology, researchers and clinicians may be able to obtain a more comprehensive, multi-layered understanding of medical images and contribute to precision medicine's armamentarium.

Radiomics and radiogenomics in gliomas: a contemporary update.

The natural history and treatment landscape of primary brain tumours are complicated by the varied tumour behaviour of primary or secondary gliomas (high-grade transformation of low-grade lesions), as well as the dilemmas with identification of radiation necrosis, tumour progression, and pseudoprogression on MRI. Radiomics and radiogenomics promise to offer precise diagnosis, predict prognosis, and assess tumour response to modern chemotherapy/immunotherapy and radiation therapy. This is achieved by a triumvirate of morphological, textural, and functional signatures, derived from a high-throughput extraction of quantitative voxel-level MR image metrics. However, the lack of standardisation of acquisition parameters and inconsistent methodology between working groups have made validations unreliable, hence multi-centre studies involving heterogenous study populations are warranted. We elucidate novel radiomic and radiogenomic workflow concepts and state-of-the-art descriptors in sub-visual MR image processing, with relevant literature on applications of such machine learning techniques in glioma management.

Test-retest repeatability of a deep learning architecture in detecting and segmenting clinically significant prostate cancer on apparent diffusion coefficient (ADC) maps.

OBJECTIVES: To evaluate short-term test-retest repeatability of a deep learning architecture (U-Net) in slice- and lesion-level detection and segmentation of clinically significant prostate cancer (csPCa: Gleason grade group > 1) using diffusion-weighted imaging fitted with monoexponential function, ADCm. METHODS: One hundred twelve patients with prostate cancer (PCa) underwent 2 prostate MRI examinations on the same day. PCa areas were annotated using whole mount prostatectomy sections. Two U-Net-based convolutional neural networks were trained on three different ADCm b value settings for (a) slice- and (b) lesion-level detection and (c) segmentation of csPCa. Short-term test-retest repeatability was estimated using intra-class correlation coefficient (ICC(3,1)), proportionate agreement, and dice similarity coefficient (DSC). A 3-fold cross-validation was performed on training set (N = 78 patients) and evaluated for performance and repeatability on testing data (N = 34 patients). RESULTS: For the three ADCm b value settings, repeatability of mean ADCm of csPCa lesions was ICC(3,1) = 0.86-0.98. Two CNNs with U-Net-based architecture demonstrated ICC(3,1) in the range of 0.80-0.83, agreement of 66-72%, and DSC of 0.68-0.72 for slice- and lesion-level detection and segmentation of csPCa. Bland-Altman plots suggest that there is no systematic bias in agreement between inter-scan ground truth segmentation repeatability and segmentation repeatability of the networks. CONCLUSIONS: For the three ADCm b value settings, two CNNs with U-Net-based architecture were repeatable for the problem of detection of csPCa at the slice-level. The network repeatability in segmenting csPCa lesions is affected by inter-scan variability and ground truth segmentation repeatability and may thus improve with better inter-scan reproducibility. KEY POINTS: • For the three ADCm b value settings, two CNNs with U-Net-based architecture were repeatable for the problem of detection of csPCa at the slice-level. • The network repeatability in segmenting csPCa lesions is affected by inter-scan variability and ground truth segmentation repeatability and may thus improve with better inter-scan reproducibility.

Normal anterior-posterior diameters of the spinal cord and spinal canal in healthy term newborns on sonography.

BACKGROUND: There are no published normal values for spinal cord and canal diameters in newborns. Spinal cord and spinal canal diameters are assessed subjectively by radiologists without any objective values for the upper limit of normal. OBJECTIVE: To determine normal values for anteroposterior (AP) diameters of the spinal cord and spinal canal on sonography in healthy term newborns. MATERIALS AND METHODS: We performed ultrasound of the entire spine on 37 healthy newborns (23 male, 14 female). The AP diameters of the spinal canal and spinal cord were measured at representative levels of the cervical (C4, C5, C6), thoracic (T5, T6, T7, T8) and lumbar spine (lumbar enlargement and above and below the lumbar enlargement level). Statistical analysis was performed to determine the mean and standard deviation of the spinal canal and spinal cord AP diameter at each aforementioned vertebral level, and their correlations with birth weight, length and head circumference. RESULTS: The mean AP spinal cord diameter was 4.1±0.5 mm at the cervical level, 3.3±0.3 mm at the thoracic level and 4.4±0.6 mm at the lumbar level. The mean AP spinal canal diameter was 7.7±0.7 mm at the cervical level, 6.2±0.8 mm at the thoracic level, and 8.4±0.7 mm at the lumbar level. CONCLUSION: In this prospective study, we have determined normal values for AP diameters of the spinal cord and spinal canal on sonography in healthy newborns at representative cervical, thoracic and lumbar levels. This data may assist in evaluating the neonatal spine in clinical situations such as suspected spinal cord injury.

Perinodular and Intranodular Radiomic Features on Lung CT Images Distinguish Adenocarcinomas from Granulomas.

Purpose To evaluate ability of radiomic (computer-extracted imaging) features to distinguish non-small cell lung cancer adenocarcinomas from granulomas at noncontrast CT. Materials and Methods For this retrospective study, screening or standard diagnostic noncontrast CT images were collected for 290 patients (mean age, 68 years; range, 18-92 years; 125 men [mean age, 67 years; range, 18-90 years] and 165 women [mean age, 68 years; range, 33-92 years]) from two institutions between 2007 and 2013. Histopathologic analysis was available for one nodule per patient. Corresponding nodule of interest was identified on axial CT images by a radiologist with manual annotation. Nodule shape, wavelet (Gabor), and texture-based (Haralick and Laws energy) features were extracted from intra- and perinodular regions. Features were pruned to train machine learning classifiers with 145 patients. In a test set of 145 patients, classifier results were compared against a convolutional neural network (CNN) and diagnostic readings of two radiologists. Results Support vector machine classifier with intranodular radiomic features achieved an area under the receiver operating characteristic curve (AUC) of 0.75 on the test set. Combining radiomics of intranodular with perinodular regions improved the AUC to 0.80. On the same test set, CNN resulted in an AUC of 0.76. Radiologist readers achieved AUCs of 0.61 and 0.60, respectively. Conclusion Radiomic features from intranodular and perinodular regions of nodules can distinguish non-small cell lung cancer adenocarcinomas from benign granulomas at noncontrast CT. © RSNA, 2018 Online supplemental material is available for this article. See also the editorial by Nishino in this issue.

Intratumoral and peritumoral radiomics for the pretreatment prediction of pathological complete response to neoadjuvant chemotherapy based on breast DCE-MRI.

BACKGROUND: In this study, we evaluated the ability of radiomic textural analysis of intratumoral and peritumoral regions on pretreatment breast cancer dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) to predict pathological complete response (pCR) to neoadjuvant chemotherapy (NAC). METHODS: A total of 117 patients who had received NAC were retrospectively analyzed. Within the intratumoral and peritumoral regions of T1-weighted contrast-enhanced MRI scans, a total of 99 radiomic textural features were computed at multiple phases. Feature selection was used to identify a set of top pCR-associated features from within a training set (n = 78), which were then used to train multiple machine learning classifiers to predict the likelihood of pCR for a given patient. Classifiers were then independently tested on 39 patients. Experiments were repeated separately among hormone receptor-positive and human epidermal growth factor receptor 2-negative (HR+, HER2-) and triple-negative or HER2+ (TN/HER2+) tumors via threefold cross-validation to determine whether receptor status-specific analysis could improve classification performance. RESULTS: Among all patients, a combined intratumoral and peritumoral radiomic feature set yielded a maximum AUC of 0.78 ± 0.030 within the training set and 0.74 within the independent testing set using a diagonal linear discriminant analysis (DLDA) classifier. Receptor status-specific feature discovery and classification enabled improved prediction of pCR, yielding maximum AUCs of 0.83 ± 0.025 within the HR+, HER2- group using DLDA and 0.93 ± 0.018 within the TN/HER2+ group using a naive Bayes classifier. In HR+, HER2- breast cancers, non-pCR was characterized by elevated peritumoral heterogeneity during initial contrast enhancement. However, TN/HER2+ tumors were best characterized by a speckled enhancement pattern within the peritumoral region of nonresponders. Radiomic features were found to strongly predict pCR independent of choice of classifier, suggesting their robustness as response predictors. CONCLUSIONS: Through a combined intratumoral and peritumoral radiomics approach, we could successfully predict pCR to NAC from pretreatment breast DCE-MRI, both with and without a priori knowledge of receptor status. Further, our findings suggest that the radiomic features most predictive of response vary across different receptor subtypes.

Radiomic features from the peritumoral brain parenchyma on treatment-naïve multi-parametric MR imaging predict long versus short-term survival in glioblastoma multiforme: Preliminary findings.

OBJECTIVE: Despite 90 % of glioblastoma (GBM) recurrences occurring in the peritumoral brain zone (PBZ), its contribution in patient survival is poorly understood. The current study leverages computerized texture (i.e. radiomic) analysis to evaluate the efficacy of PBZ features from pre-operative MRI in predicting long- (>18 months) versus short-term (<7 months) survival in GBM. METHODS: Sixty-five patient examinations (29 short-term, 36 long-term) with gadolinium-contrast T1w, FLAIR and T2w sequences from the Cancer Imaging Archive were employed. An expert manually segmented each study as: enhancing lesion, PBZ and tumour necrosis. 402 radiomic features (capturing co-occurrence, grey-level dependence and directional gradients) were obtained for each region. Evaluation was performed using threefold cross-validation, such that a subset of studies was used to select the most predictive features, and the remaining subset was used to evaluate their efficacy in predicting survival. RESULTS: A subset of ten radiomic 'peritumoral' MRI features, suggestive of intensity heterogeneity and textural patterns, was found to be predictive of survival (p = 1.47 × 10-5) as compared to features from enhancing tumour, necrotic regions and known clinical factors. CONCLUSION: Our preliminary analysis suggests that radiomic features from the PBZ on routine pre-operative MRI may be predictive of long- versus short-term survival in GBM. KEY POINTS: • Radiomic features from peritumoral regions can capture glioblastoma heterogeneity to predict outcome. • Peritumoral radiomics along with clinical factors are highly predictive of glioblastoma outcome. • Identifying prognostic markers can assist in making personalized therapy decisions in glioblastoma.

PathLDM: Text conditioned Latent Diffusion Model for Histopathology.

To achieve high-quality results, diffusion models must be trained on large datasets. This can be notably prohibitive for models in specialized domains, such as computational pathology. Conditioning on labeled data is known to help in data-efficient model training. Therefore, histopathology reports, which are rich in valuable clinical information, are an ideal choice as guidance for a histopathology generative model. In this paper, we introduce PathLDM, the first text-conditioned Latent Diffusion Model tailored for generating high-quality histopathology images. Leveraging the rich contextual information provided by pathology text reports, our approach fuses image and textual data to enhance the generation process. By utilizing GPT's capabilities to distill and summarize complex text reports, we establish an effective conditioning mechanism. Through strategic conditioning and necessary architectural enhancements, we achieved a SoTA FID score of 7.64 for text-to-image generation on the TCGA-BRCA dataset, significantly outperforming the closest text-conditioned competitor with FID 30.1.

-New frontiers in domain-inspired radiomics and radiogenomics: increasing role of molecular diagnostics in CNS tumor classification and grading following WHO CNS-5 updates.

Gliomas and Glioblastomas represent a significant portion of central nervous system (CNS) tumors associated with high mortality rates and variable prognosis. In 2021, the World Health Organization (WHO) updated its Glioma classification criteria, most notably incorporating molecular markers including CDKN2A/B homozygous deletion, TERT promoter mutation, EGFR amplification, + 7/-10 chromosome copy number changes, and others into the grading and classification of adult and pediatric Gliomas. The inclusion of these markers and the corresponding introduction of new Glioma subtypes has allowed for more specific tailoring of clinical interventions and has inspired a new wave of Radiogenomic studies seeking to leverage medical imaging information to explore the diagnostic and prognostic implications of these new biomarkers. Radiomics, deep learning, and combined approaches have enabled the development of powerful computational tools for MRI analysis correlating imaging characteristics with various molecular biomarkers integrated into the updated WHO CNS-5 guidelines. Recent studies have leveraged these methods to accurately classify Gliomas in accordance with these updated molecular-based criteria based solely on non-invasive MRI, demonstrating the great promise of Radiogenomic tools. In this review, we explore the relative benefits and drawbacks of these computational frameworks and highlight the technical and clinical innovations presented by recent studies in the landscape of fast evolving molecular-based Glioma subtyping. Furthermore, the potential benefits and challenges of incorporating these tools into routine radiological workflows, aiming to enhance patient care and optimize clinical outcomes in the evolving field of CNS tumor management, have been highlighted.

Attention De-sparsification Matters: Inducing diversity in digital pathology representation learning.

We propose DiRL, a Diversity-inducing Representation Learning technique for histopathology imaging. Self-supervised learning (SSL) techniques, such as contrastive and non-contrastive approaches, have been shown to learn rich and effective representations of digitized tissue samples with limited pathologist supervision. Our analysis of vanilla SSL-pretrained models' attention distribution reveals an insightful observation: sparsity in attention, i.e, models tends to localize most of their attention to some prominent patterns in the image. Although attention sparsity can be beneficial in natural images due to these prominent patterns being the object of interest itself, this can be sub-optimal in digital pathology; this is because, unlike natural images, digital pathology scans are not object-centric, but rather a complex phenotype of various spatially intermixed biological components. Inadequate diversification of attention in these complex images could result in crucial information loss. To address this, we leverage cell segmentation to densely extract multiple histopathology-specific representations, and then propose a prior-guided dense pretext task, designed to match the multiple corresponding representations between the views. Through this, the model learns to attend to various components more closely and evenly, thus inducing adequate diversification in attention for capturing context-rich representations. Through quantitative and qualitative analysis on multiple tasks across cancer types, we demonstrate the efficacy of our method and observe that the attention is more globally distributed.

TopoTxR: A topology-guided deep convolutional network for breast parenchyma learning on DCE-MRIs.

Characterization of breast parenchyma in dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) is a challenging task owing to the complexity of underlying tissue structures. Existing quantitative approaches, like radiomics and deep learning models, lack explicit quantification of intricate and subtle parenchymal structures, including fibroglandular tissue. To address this, we propose a novel topological approach that explicitly extracts multi-scale topological structures to better approximate breast parenchymal structures, and then incorporates these structures into a deep-learning-based prediction model via an attention mechanism. Our topology-informed deep learning model, TopoTxR, leverages topology to provide enhanced insights into tissues critical for disease pathophysiology and treatment response. We empirically validate TopoTxR using the VICTRE phantom breast dataset, showing that the topological structures extracted by our model effectively approximate the breast parenchymal structures. We further demonstrate TopoTxR's efficacy in predicting response to neoadjuvant chemotherapy. Our qualitative and quantitative analyses suggest differential topological behavior of breast tissue in treatment-naïve imaging, in patients who respond favorably to therapy as achieving pathological complete response (pCR) versus those who do not. In a comparative analysis with several baselines on the publicly available I-SPY 1 dataset (N = 161, including 47 patients with pCR and 114 without) and the Rutgers proprietary dataset (N = 120, with 69 patients achieving pCR and 51 not), TopoTxR demonstrates a notable improvement, achieving a 2.6% increase in accuracy and a 4.6% enhancement in AUC compared to the state-of-the-art method.

Automated Detection of Cervical Spinal Stenosis and Cord Compression via Vision Transformer and Rules-Based Classification.

BACKGROUND AND PURPOSE: Cervical spinal cord compression, defined as spinal cord deformity and severe narrowing of the spinal canal in the cervical region, can lead to severe clinical consequences, including intractable pain, sensory disturbance, paralysis, and even death, and may require emergent intervention to prevent negative outcomes. Despite the critical nature of cord compression, no automated tool is available to alert clinical radiologists to the presence of such findings. This study aims to demonstrate the ability of a vision transformer (ViT) model for the accurate detection of cervical cord compression. MATERIALS AND METHODS: A clinically diverse cohort of 142 cervical spine MRIs was identified, 34% of which were normal or had mild stenosis, 31% with moderate stenosis, and 35% with cord compression. Utilizing gradient-echo images, slices were labeled as no cord compression/mild stenosis, moderate stenosis, or severe stenosis/cord compression. Segmentation of the spinal canal was performed and confirmed by neuroradiology faculty. A pretrained ViT model was fine-tuned to predict section-level severity by using a train:validation:test split of 60:20:20. Each examination was assigned an overall severity based on the highest level of section severity, with an examination labeled as positive for cord compression if ≥1 section was predicted in the severe category. Additionally, 2 convolutional neural network (CNN) models (ResNet50, DenseNet121) were tested in the same manner. RESULTS: The ViT model outperformed both CNN models at the section level, achieving section-level accuracy of 82%, compared with 72% and 78% for ResNet and DenseNet121, respectively. ViT patient-level classification achieved accuracy of 93%, sensitivity of 0.90, positive predictive value of 0.90, specificity of 0.95, and negative predictive value of 0.95. Receiver operating characteristic area under the curve was greater for ViT than either CNN. CONCLUSIONS: This classification approach using a ViT model and rules-based classification accurately detects the presence of cervical spinal cord compression at the patient level. In this study, the ViT model outperformed both conventional CNN approaches at the section and patient levels. If implemented into the clinical setting, such a tool may streamline neuroradiology workflow, improving efficiency and consistency.

Performance of GPT-4 on the American College of Radiology In-training Examination: Evaluating Accuracy, Model Drift, and Fine-tuning.

RATIONALE AND OBJECTIVES: In our study, we evaluate GPT-4's performance on the American College of Radiology (ACR) 2022 Diagnostic Radiology In-Training Examination (DXIT). We perform multiple experiments across time points to assess for model drift, as well as after fine-tuning to assess for differences in accuracy. MATERIALS AND METHODS: Questions were sequentially input into GPT-4 with a standardized prompt. Each answer was recorded and overall accuracy was calculated, as was logic-adjusted accuracy, and accuracy on image-based questions. This experiment was repeated several months later to assess for model drift, then again after the performance of fine-tuning to assess for changes in GPT's performance. RESULTS: GPT-4 achieved 58.5% overall accuracy, lower than the PGY-3 average (61.9%) but higher than the PGY-2 average (52.8%). Adjusted accuracy was 52.8%. GPT-4 showed significantly higher (p = 0.012) confidence for correct answers (87.1%) compared to incorrect (84.0%). Performance on image-based questions was significantly poorer (p < 0.001) at 45.4% compared to text-only questions (80.0%), with adjusted accuracy for image-based questions of 36.4%. When the questions were repeated, GPT-4 chose a different answer 25.5% of the time and there was no change in accuracy. Fine-tuning did not improve accuracy. CONCLUSION: GPT-4 performed between PGY-2 and PGY-3 levels on the 2022 DXIT, significantly poorer on image-based questions, and with large variability in answer choices across time points. Exploratory experiments in fine-tuning did not improve performance. This study underscores the potential and risks of using minimally-prompted general AI models in interpreting radiologic images as a diagnostic tool. Implementers of general AI radiology systems should exercise caution given the possibility of spurious yet confident responses.

Pretreatment Spatially Aware Magnetic Resonance Imaging Radiomics Can Predict Distant Brain Metastases (DBMs) After Stereotactic Radiosurgery/Radiation Therapy (SRS/SRT).

Purpose: Stereotactic radiosurgery/radiation therapy (SRS/SRT) increasingly has been used to treat brain metastases. However, the development of distant brain metastases (DBMs) in the untreated brain remains a serious complication. We sought to develop a spatially aware radiomic signature to model the time-to-DBM development in a cohort of patients leveraging pretreatment magnetic resonance imaging (MRI) and radiation therapy treatment planning data including radiation dose distribution maps. Methods and Materials: We retrospectively analyzed a cohort of 105 patients with brain metastases treated by SRS/SRT with pretreatment multiparametric MRI (T1, T1 postcontrast, T2, fluid-attenuated inversion recovery). Three-dimensional radiomic features were extracted from each MRI sequence within 5 isodose regions of interest (ROIs) identified via radiation dose distribution maps and gross target volume (GTV) contours. Clinical features including patient performance status, number of lesions treated, tumor volume, and tumor stage were collected to serve as a baseline for comparison. Cox proportional hazards (CPH) modeling and Kaplan-Meier analysis were used to model time-to-DBM development. Results: CPH models trained using radiomic features achieved a mean concordance index (c-index) of 0.63 (standard deviation [SD], 0.08) compared with a c-index of 0.49 (SD, 0.09) for CPH models trained using clinical factors. A CPH model trained using both radiomic and clinical features achieved a c-index of 0.69 (SD, 0.08). The identified radiomic signature was able to stratify patients into distinct risk groups with statistically significant differences (P = .00007) in time-to-DBM development as measured by log-rank test. Clinical features were unable to do the same. Radiomic features from the peritumoral 50% to 75% isodose ROI and GTV region were most predictive of DBM development. Conclusions: Our results suggest that radiomic features extracted from pretreatment MRI and multiple isodose ROIs can model time-to-DBM development in patients receiving SRS/SRT for brain metastases, outperforming clinical feature baselines. Notably, we believe we are the first to leverage SRS/SRT dose maps for ROI identification and subsequent radiomic analysis of peritumoral and untargeted brain regions using multiparametric MRI. We observed that the peritumoral environment may be implicated in DBM development for SRS/SRT-treated brain metastases. Our preliminary results might enable the identification of patients with predisposition to DBM development and prompt subsequent changes in disease management.

Eco-evolutionary Guided Pathomic Analysis to Predict DCIS Upstaging.

Cancers evolve in a dynamic ecosystem. Thus, characterizing cancer's ecological dynamics is crucial to understanding cancer evolution and can lead to discovering novel biomarkers to predict disease progression. Ductal carcinoma in situ (DCIS) is an early-stage breast cancer characterized by abnormal epithelial cell growth confined within the milk ducts. Although there has been extensive research on genetic and epigenetic causes of breast carcinogenesis, none of these studies have successfully identified a biomarker for the progression and/or upstaging of DCIS. In this study, we show that ecological habitat analysis of hypoxia and acidosis biomarkers can significantly improve prediction of DCIS upstaging. First, we developed a novel eco-evolutionary designed approach to define habitats in the tumor intra-ductal microenvironment based on oxygen diffusion distance in our DCIS cohort of 84 patients. Then, we identify cancer cells with metabolic phenotypes attributed to their habitat conditions, such as the expression of CA9 indicating hypoxia responding phenotype, and LAMP2b indicating a hypoxia-induced acid adaptation. Traditionally these markers have shown limited predictive capabilities for DCIS upstaging, if any. However, when analyzed from an ecological perspective, their power to differentiate between indolent and upstaged DCIS increased significantly. Second, using eco-evolutionary guided computational and digital pathology techniques, we discovered distinct spatial patterns of these biomarkers and used the distribution of such patterns to predict patient upstaging. The patterns were characterized by both cellular features and spatial features. With a 5-fold validation on the biopsy cohort, we trained a random forest classifier to achieve the area under curve(AUC) of 0.74. Our results affirm the importance of using eco-evolutionary-designed approaches in biomarkers discovery studies in the era of digital pathology by demonstrating the role of eco-evolution dynamics in predicting cancer progression.

Convolutional Neural Networks (CNNs) for Pneumonia Classification on Pediatric Chest Radiographs.

BACKGROUND:  Pneumonia is an infectious disease that is especially harmful to those with weak immune systems, such as children under the age of 5. While radiologists' diagnosis of pediatric pneumonia on chest radiographs (CXRs) is often accurate, subtle findings can be missed due to the subjective nature of the diagnosis process. Artificial intelligence (AI) techniques, such as convolutional neural networks (CNNs), can help make the process more objective and precise. However, off-the-shelf CNNs may perform poorly if they are not tuned to their appropriate hyperparameters. Our study aimed to identify the CNNs and their hyperparameter combinations (dropout, batch size, and optimizer) that optimize model performance. METHODOLOGY:  Sixty models based on five CNNs (VGG 16, VGG 19, DenseNet 121, DenseNet 169, and InceptionResNet V2) and 12 hyperparameter combinations were tested. Adam, Root Mean Squared Propagation (RmsProp), and Mini-Batch Stochastic Gradient Descent (SGD) optimizers were used. Two batch sizes, 32 and 64, were utilized. A dropout rate of either 0.5 or 0.7 was used in all dropout layers. We used a deidentified CXR dataset of 4200 pneumonia (Figure 1a) and 1600 normal images (Figure 1b). Seventy percent of the CXRs in the dataset were used for training the model, 20% were used for validating the model, and 10% were used for testing the model. All CNNs were trained first on the ImageNet dataset. They were then trained, with frozen weights, on the CXR-containing dataset.  Results: Among the 60 models, VGG-19 (dropout of 0.5, batch size of 32, and Adam optimizer) was the most accurate. This model achieved an accuracy of 87.9%. A dropout of 0.5 consistently gave higher accuracy, area under the receiver operating characteristics curve (AUROC), and area under the precision-recall curve (AUPRC) compared to a dropout of 0.7. The CNNs InceptionResNet V2, DenseNet 169, VGG 16, and VGG 19 significantly outperformed the DenseNet121 CNN in accuracy and AUROC. The Adam and RmsProp optimizer had improved AUROC and AUPRC compared to the SGD optimizer. The batch size had no statistically significant effect on model performance. CONCLUSION:  We recommend using low dropout rates (0.5) and RmsProp or Adam optimizer for pneumonia-detecting CNNs. Additionally, we discourage using the DenseNet121 CNN when other CNNs are available. Finally, the batch size may be set to any value, dependent on computational resources.

Enhancing Modality-Agnostic Representations via Meta-learning for Brain Tumor Segmentation.

In medical vision, different imaging modalities provide complementary information. However, in practice, not all modalities may be available during inference or even training. Previous approaches, e.g., knowledge distillation or image synthesis, often assume the availability of full modalities for all subjects during training; this is unrealistic and impractical due to the variability in data collection across sites. We propose a novel approach to learn enhanced modality-agnostic representations by employing a meta-learning strategy in training, even when only limited full modality samples are available. Meta-learning enhances partial modality representations to full modality representations by meta-training on partial modality data and meta-testing on limited full modality samples. Additionally, we co-supervise this feature enrichment by introducing an auxiliary adversarial learning branch. More specifically, a missing modality detector is used as a discriminator to mimic the full modality setting. Our segmentation framework significantly outperforms state-of-the-art brain tumor segmentation techniques in missing modality scenarios.

Token Sparsification for Faster Medical Image Segmentation.

Can we use sparse tokens for dense prediction, e.g., segmentation? Although token sparsification has been applied to Vision Transformers (ViT) to accelerate classification, it is still unknown how to perform segmentation from sparse tokens. To this end, we reformulate segmentation as a sparse encoding → token completion → dense decoding (SCD) pipeline. We first empirically show that naïvely applying existing approaches from classification token pruning and masked image modeling (MIM) leads to failure and inefficient training caused by inappropriate sampling algorithms and the low quality of the restored dense features. In this paper, we propose Soft-topK Token Pruning (STP) and Multi-layer Token Assembly (MTA) to address these problems. In sparse encoding, STP predicts token importance scores with a lightweight sub-network and samples the topK tokens. The intractable topK gradients are approximated through a continuous perturbed score distribution. In token completion, MTA restores a full token sequence by assembling both sparse output tokens and pruned multi-layer intermediate ones. The last dense decoding stage is compatible with existing segmentation decoders, e.g., UNETR. Experiments show SCD pipelines equipped with STP and MTA are much faster than baselines without token pruning in both training (up to 120% higher throughput) and inference (up to 60.6% higher throughput) while maintaining segmentation quality. Code is available here: https://github.com/cvlab-stonybrook/TokenSparse-for-MedSeg.