DPACFuse: Dual-Branch Progressive Learning for Infrared and Visible Image Fusion with Complementary Self-Attention and Convolution.

Infrared and visible image fusion aims to generate a single fused image that not only contains rich texture details and salient objects, but also facilitates downstream tasks. However, existing works mainly focus on learning different modality-specific or shared features, and ignore the importance of modeling cross-modality features. To address these challenges, we propose Dual-branch Progressive learning for infrared and visible image fusion with a complementary self-Attention and Convolution (DPACFuse) network. On the one hand, we propose Cross-Modality Feature Extraction (CMEF) to enhance information interaction and the extraction of common features across modalities. In addition, we introduce a high-frequency gradient convolution operation to extract fine-grained information and suppress high-frequency information loss. On the other hand, to alleviate the CNN issues of insufficient global information extraction and computation overheads of self-attention, we introduce the ACmix, which can fully extract local and global information in the source image with a smaller computational overhead than pure convolution or pure self-attention. Extensive experiments demonstrated that the fused images generated by DPACFuse not only contain rich texture information, but can also effectively highlight salient objects. Additionally, our method achieved approximately 3% improvement over the state-of-the-art methods in MI, Qabf, SF, and AG evaluation indicators. More importantly, our fused images enhanced object detection and semantic segmentation by approximately 10%, compared to using infrared and visible images separately.

An artificial intelligence approach to classify pathogenic fungal genera of fungal keratitis using corneal confocal microscopy images.

PURPOSE: Fungal keratitis is a common cause of blindness worldwide. Timely identification of the causative fungal genera is essential for clinical management. In vivo confocal microscopy (IVCM) provides useful information on pathogenic genera. This study attempted to apply deep learning (DL) to establish an automated method to identify pathogenic fungal genera using IVCM images. METHODS: Deep learning networks were trained, validated, and tested using a data set of 3364 IVCM images that collected from 100 eyes of 100 patients with culture-proven filamentous fungal keratitis. Two transfer learning approaches were investigated: one was a combined framework that extracted features by a DL network and adopted decision tree (DT) as a classifier; another was a complete supervised DL model which used DL-based fully connected layers to implement the classification. RESULTS: The DL classifier model revealed better performance compared with the DT classifier model in an independent testing set. The DL classifier model showed an area under the receiver operating characteristic curves (AUC) of 0.887 with an accuracy of 0.817, sensitivity of 0.791, specificity of 0.831, G-mean of 0.811, and F1 score of 0.749 in identifying Fusarium, and achieved an AUC of 0.827 with an accuracy of 0.757, sensitivity of 0.756, specificity of 0.759, G-mean of 0.757, and F1 score of 0.716 in identifying Aspergillus. CONCLUSION: The DL model can classify Fusarium and Aspergillus by learning effective features in IVCM images automatically. The automated IVCM image analysis suggests a noninvasive identification of Fusarium and Aspergillus with clear potential application in early diagnosis and management of fungal keratitis.

The Crohn's-like lymphoid reaction density: a new artificial intelligence quantified prognostic immune index in colon cancer.

BACKGROUND: The Crohn's-like lymphoid reaction (CLR) is manifested as peritumoral lymphocytes aggregation in colon cancer, which is a major component of the host immune response to cancer. However, the lack of a unified and objective CLR evaluation standard limits its clinical application. We, therefore, developed a deep learning model for the fully automated CLR density quantification on routine hematoxylin and eosin (HE)-stained whole-slide images (WSIs) and further investigated its prognostic validity for patient stratification. METHODS: The CLR density was calculated by using a deep learning method on HE-stained WSIs. A training (N = 279) and a validation (N = 194) cohorts were used to evaluate the prognostic value of CLR density for overall survival (OS). RESULT: The fully automated quantified CLR density was an independent prognostic factor, with high CLR density associated with increased OS in the discovery (HR 0.58, 95% CI 0.38-0.89, P = 0.012) and validation cohort (0.45, 0.23-0.88, 0.020). Integrating CLR density into a Cox model with other risk factors showed improved prognostic capability. CONCLUSION: We developed a new immune indicator (CLR density) quantified by a deep learning method to evaluate the lymphocytes aggregation in colon cancer. The CLR density was demonstrated its predictive value for OS in two independent cohorts. This approach allows for the objective and standardized quantification while reducing pathologists' workload. Therefore, this fully automated standardized method of CLR evaluation had potential clinical value.

Multi-scale pathology image texture signature is a prognostic factor for resectable lung adenocarcinoma: a multi-center, retrospective study.

BACKGROUND: Tumor histomorphology analysis plays a crucial role in predicting the prognosis of resectable lung adenocarcinoma (LUAD). Computer-extracted image texture features have been previously shown to be correlated with outcome. However, a comprehensive, quantitative, and interpretable predictor remains to be developed. METHODS: In this multi-center study, we included patients with resectable LUAD from four independent cohorts. An automated pipeline was designed for extracting texture features from the tumor region in hematoxylin and eosin (H&E)-stained whole slide images (WSIs) at multiple magnifications. A multi-scale pathology image texture signature (MPIS) was constructed with the discriminative texture features in terms of overall survival (OS) selected by the LASSO method. The prognostic value of MPIS for OS was evaluated through univariable and multivariable analysis in the discovery set (n = 111) and the three external validation sets (V1, n = 115; V2, n = 116; and V3, n = 246). We constructed a Cox proportional hazards model incorporating clinicopathological variables and MPIS to assess whether MPIS could improve prognostic stratification. We also performed histo-genomics analysis to explore the associations between texture features and biological pathways. RESULTS: A set of eight texture features was selected to construct MPIS. In multivariable analysis, a higher MPIS was associated with significantly worse OS in the discovery set (HR 5.32, 95%CI 1.72-16.44; P = 0.0037) and the three external validation sets (V1: HR 2.63, 95%CI 1.10-6.29, P = 0.0292; V2: HR 2.99, 95%CI 1.34-6.66, P = 0.0075; V3: HR 1.93, 95%CI 1.15-3.23, P = 0.0125). The model that integrated clinicopathological variables and MPIS had better discrimination for OS compared to the clinicopathological variables-based model in the discovery set (C-index, 0.837 vs. 0.798) and the three external validation sets (V1: 0.704 vs. 0.679; V2: 0.728 vs. 0.666; V3: 0.696 vs. 0.669). Furthermore, the identified texture features were associated with biological pathways, such as cytokine activity, structural constituent of cytoskeleton, and extracellular matrix structural constituent. CONCLUSIONS: MPIS was an independent prognostic biomarker that was robust and interpretable. Integration of MPIS with clinicopathological variables improved prognostic stratification in resectable LUAD and might help enhance the quality of individualized postoperative care.

Automated whole-slide images assessment of immune infiltration in resected non-small-cell lung cancer: towards better risk-stratification.

BACKGROUND: High immune infiltration is associated with favourable prognosis in patients with non-small-cell lung cancer (NSCLC), but an automated workflow for characterizing immune infiltration, with high validity and reliability, remains to be developed. METHODS: We performed a multicentre retrospective study of patients with completely resected NSCLC. We developed an image analysis workflow for automatically evaluating the density of CD3+ and CD8+ T-cells in the tumour regions on immunohistochemistry (IHC)-stained whole-slide images (WSIs), and proposed an immune scoring system "I-score" based on the automated assessed cell density. RESULTS: A discovery cohort (n = 145) and a validation cohort (n = 180) were used to assess the prognostic value of the I-score for disease-free survival (DFS). The I-score (two-category) was an independent prognostic factor after adjusting for other clinicopathologic factors. Compared with a low I-score (two-category), a high I-score was associated with significantly superior DFS in the discovery cohort (adjusted hazard ratio [HR], 0.54; 95% confidence interval [CI] 0.33-0.86; P = 0.010) and validation cohort (adjusted HR, 0.57; 95% CI 0.36-0.92; P = 0.022). The I-score improved the prognostic stratification when integrating it into the Cox proportional hazard regression models with other risk factors (discovery cohort, C-index 0.742 vs. 0.728; validation cohort, C-index 0.695 vs. 0.685). CONCLUSION: This automated workflow and immune scoring system would advance the clinical application of immune microenvironment evaluation and support the clinical decision making for patients with resected NSCLC.

MRI characteristics of breast edema for assessing axillary lymph node burden in early-stage breast cancer: a retrospective bicentric study.

OBJECTIVES: To investigate whether breast edema characteristics at preoperative T2-weighted imaging (T2WI) could help evaluate axillary lymph node (ALN) burden in patients with early-stage breast cancer. METHODS: This retrospective study included women with clinical T1 and T2 stage breast cancer and preoperative MRI examination in two independent cohorts from May 2014 to December 2020. Low (< 3 LNs+) and high (≥ 3 LNs+) pathological ALN (pALN) burden were recorded as endpoint. Breast edema score (BES) was evaluated at T2WI. Univariable and multivariable analyses were performed by the logistic regression model. The added predictive value of BES was examined utilizing the area under the curve (AUC), net reclassification improvement (NRI), and integrated discrimination improvement (IDI). RESULTS: A total of 1092 patients were included in this study. BES was identified as the independent predictor of pALN burden in primary (n = 677) and validation (n = 415) cohorts. The analysis using MRI-ALN status showed that BES significantly improved the predictive performance of pALN burden (AUC: 0.65 vs 0.71, p < 0.001; IDI = 0.045, p < 0.001; continuous NRI = 0.159, p = 0.050). These results were confirmed in the validation cohort (AUC: 0.64 vs 0.69, p = 0.009; IDI = 0.050, p < 0.001; continuous NRI = 0.213, p = 0.047). Furthermore, BES was positively correlated with biologically invasive clinicopathological factors (p < 0.05). CONCLUSIONS: In individuals with early-stage breast cancer, preoperative MRI characteristics of breast edema could be a promising predictor for pALN burden, which may aid in treatment planning. KEY POINTS: • In this retrospective study of 1092 patients with early-stage breast cancer from two cohorts, the MRI characteristic of breast edema has independent and additive predictive value for assessing axillary lymph node burden. • Breast edema characteristics at T2WI positively correlated with biologically invasive clinicopathological factors, which may be useful for preoperative diagnosis and treatment planning for individual patients with breast cancer.

FDMLNet: A Frequency-Division and Multiscale Learning Network for Enhancing Low-Light Image.

Low-illumination images exhibit low brightness, blurry details, and color casts, which present us an unnatural visual experience and further have a negative effect on other visual applications. Data-driven approaches show tremendous potential for lighting up the image brightness while preserving its visual naturalness. However, these methods introduce hand-crafted holes and noise enlargement or over/under enhancement and color deviation. For mitigating these challenging issues, this paper presents a frequency division and multiscale learning network named FDMLNet, including two subnets, DetNet and StruNet. This design first applies the guided filter to separate the high and low frequencies of authentic images, then DetNet and StruNet are, respectively, developed to process them, to fully explore their information at different frequencies. In StruNet, a feasible feature extraction module (FFEM), grouped by multiscale learning block (MSL) and a dual-branch channel attention mechanism (DCAM), is injected to promote its multiscale representation ability. In addition, three FFEMs are connected in a new dense connectivity meant to utilize multilevel features. Extensive quantitative and qualitative experiments on public benchmarks demonstrate that our FDMLNet outperforms state-of-the-art approaches benefiting from its stronger multiscale feature expression and extraction ability.

A deep learning quantified stroma-immune score to predict survival of patients with stage II-III colorectal cancer.

BACKGROUND: Profound heterogeneity in prognosis has been observed in colorectal cancer (CRC) patients with intermediate levels of disease (stage II-III), advocating the identification of valuable biomarkers that could improve the prognostic stratification. This study aims to develop a deep learning-based pipeline for fully automatic quantification of immune infiltration within the stroma region on immunohistochemical (IHC) whole-slide images (WSIs) and further analyze its prognostic value in CRC. METHODS: Patients from two independent cohorts were divided into three groups: the development group (N = 200), the internal (N = 134), and the external validation group (N = 90). We trained a convolutional neural network for tissue classification of CD3 and CD8 stained WSIs. A scoring system, named stroma-immune score, was established by quantifying the density of CD3+ and CD8+ T-cells infiltration in the stroma region. RESULTS: Patients with higher stroma-immune scores had much longer survival. In the development group, 5-year survival rates of the low and high scores were 55.7% and 80.8% (hazard ratio [HR] for high vs. low 0.39, 95% confidence interval [CI] 0.24-0.63, P < 0.001). These results were confirmed in the internal and external validation groups with 5-year survival rates of low and high scores were 57.1% and 78.8%, 63.9% and 88.9%, respectively (internal: HR for high vs. low 0.49, 95% CI 0.28-0.88, P = 0.017; external: HR for high vs. low 0.35, 95% CI 0.15-0.83, P = 0.018). The combination of stroma-immune score and tumor-node-metastasis (TNM) stage showed better discrimination ability for survival prediction than using the TNM stage alone. CONCLUSIONS: We proposed a stroma-immune score via a deep learning-based pipeline to quantify CD3+ and CD8+ T-cells densities within the stroma region on WSIs of CRC and further predict survival.

Identification of Multi-Class Drugs Based on Near Infrared Spectroscopy and Bidirectional Generative Adversarial Networks.

Drug detection and identification technology are of great significance in drug supervision and management. To determine the exact source of drugs, it is often necessary to directly identify multiple varieties of drugs produced by multiple manufacturers. Near-infrared spectroscopy (NIR) combined with chemometrics is generally used in these cases. However, existing NIR classification modeling methods have great limitations in dealing with a large number of categories and spectra, especially under the premise of insufficient samples, unbalanced samples, and sensitive identification error cost. Therefore, this paper proposes a NIR multi-classification modeling method based on a modified Bidirectional Generative Adversarial Networks (Bi-GAN). It makes full utilization of the powerful feature extraction ability and good sample generation quality of Bi-GAN and uses the generated samples with obvious features, an equal number between classes, and a sufficient number within classes to replace the unbalanced and insufficient real samples in the courses of spectral classification. 1721 samples of four kinds of drugs produced by 29 manufacturers were used as experimental materials, and the results demonstrate that this method is superior to other comparative methods in drug NIR classification scenarios, and the optimal accuracy rate is even more than 99% under ideal conditions.

PA-Net: A phase attention network fusing venous and arterial phase features of CT images for liver tumor segmentation.

BACKGROUND AND OBJECTIVE: Liver cancer seriously threatens human health. In clinical diagnosis, contrast-enhanced computed tomography (CECT) images provide important supplementary information for accurate liver tumor segmentation. However, most of the existing methods of liver tumor automatic segmentation focus only on single-phase image features. And the existing multi-modal methods have limited segmentation effect due to the redundancy of fusion features. In addition, the spatial misalignment of multi-phase images causes feature interference. METHODS: In this paper, we propose a phase attention network (PA-Net) to adequately aggregate multi-phase information of CT images and improve segmentation performance for liver tumors. Specifically, we design a PA module to generate attention weight maps voxel by voxel to efficiently fuse multi-phase CT images features to avoid feature redundancy. In order to solve the problem of feature interference in the multi-phase image segmentation task, we design a new learning strategy and prove its effectiveness experimentally. RESULTS: We conduct comparative experiments on the in-house clinical dataset and achieve the SOTA segmentation performance on multi-phase methods. In addition, our method has improved the mean dice score by 3.3% compared with the single-phase method based on nnUNet, and our learning strategy has improved the mean dice score by 1.51% compared with the ML strategy. CONCLUSION: The experimental results show that our method is superior to the existing multi-phase liver tumor segmentation method, and provides a scheme for dealing with missing modalities in multi-modal tasks. In addition, our proposed learning strategy makes more effective use of arterial phase image information and is proven to be the most effective in liver tumor segmentation tasks using thick-layer CT images. The source code is released on (https://github.com/Houjunfeng203934/PA-Net).

ERNet: Edge Regularization Network for Cerebral Vessel Segmentation in Digital Subtraction Angiography Images.

Stroke is a leading cause of disability and fatality in the world, with ischemic stroke being the most common type. Digital Subtraction Angiography images, the gold standard in the operation process, can accurately show the contours and blood flow of cerebral vessels. The segmentation of cerebral vessels in DSA images can effectively help physicians assess the lesions. However, due to the disturbances in imaging parameters and changes in imaging scale, accurate cerebral vessel segmentation in DSA images is still a challenging task. In this paper, we propose a novel Edge Regularization Network (ERNet) to segment cerebral vessels in DSA images. Specifically, ERNet employs the erosion and dilation processes on the original binary vessel annotation to generate pseudo-ground truths of False Negative and False Positive, which serve as constraints to refine the coarse predictions based on their mapping relationship with the original vessels. In addition, we exploit a Hybrid Fusion Module based on convolution and transformers to extract local features and build long-range dependencies. Moreover, to support and advance the open research in the field of ischemic stroke, we introduce FPDSA, the first pixel-level semantic segmentation dataset for cerebral vessels. Extensive experiments on FPDSA illustrate the leading performance of our ERNet.

Spatial distance between tumor and lymphocyte can predict the survival of patients with resectable lung adenocarcinoma.

Background and objective: Spatial interaction between tumor-infiltrating lymphocytes (TILs) and tumor cells is valuable in predicting the effectiveness of immune response and prognosis amongst patients with lung adenocarcinoma (LUAD). Recent evidence suggests that the spatial distance between tumor cells and lymphocytes also influences the immune responses, but the distance analysis based on Hematoxylin and Eosin (H&E) -stained whole-slide images (WSIs) remains insufficient. To address this issue, we aim to explore the relationship between distance and prognosis prediction of patients with LUAD in this study. Methods: We recruited patients with resectable LUAD from three independent cohorts in this multi-center study. We proposed a simple but effective deep learning-driven workflow to automatically segment different cell types in the tumor region using the HoVer-Net model, and quantified the spatial distance (DIST) between tumor cells and lymphocytes based on H&E-stained WSIs. The association of DIST with disease-free survival (DFS) was explored in the discovery set (D1, n = 276) and the two validation sets (V1, n = 139; V2, n = 115). Results: In multivariable analysis, the low DIST group was associated with significantly better DFS in the discovery set (D1, HR, 0.61; 95 % CI, 0.40-0.94; p = 0.027) and the two validation sets (V1, HR, 0.54; 95 % CI, 0.32-0.91; p = 0.022; V2, HR, 0.44; 95 % CI, 0.24-0.81; p = 0.009). By integrating the DIST with clinicopathological factors, the integrated model (full model) had better discrimination for DFS in the discovery set (C-index, D1, 0.745 vs. 0.723) and the two validation sets (V1, 0.621 vs. 0.596; V2, 0.671 vs. 0.650). Furthermore, the computerized DIST was associated with immune phenotypes such as immune-desert and inflamed phenotypes. Conclusions: The integration of DIST with clinicopathological factors could improve the stratification performance of patients with resectable LUAD, was beneficial for the prognosis prediction of LUAD patients, and was also expected to assist physicians in individualized treatment.

DIAS: A dataset and benchmark for intracranial artery segmentation in DSA sequences.

The automated segmentation of Intracranial Arteries (IA) in Digital Subtraction Angiography (DSA) plays a crucial role in the quantification of vascular morphology, significantly contributing to computer-assisted stroke research and clinical practice. Current research primarily focuses on the segmentation of single-frame DSA using proprietary datasets. However, these methods face challenges due to the inherent limitation of single-frame DSA, which only partially displays vascular contrast, thereby hindering accurate vascular structure representation. In this work, we introduce DIAS, a dataset specifically developed for IA segmentation in DSA sequences. We establish a comprehensive benchmark for evaluating DIAS, covering full, weak, and semi-supervised segmentation methods. Specifically, we propose the vessel sequence segmentation network, in which the sequence feature extraction module effectively captures spatiotemporal representations of intravascular contrast, achieving intracranial artery segmentation in 2D+Time DSA sequences. For weakly-supervised IA segmentation, we propose a novel scribble learning-based image segmentation framework, which, under the guidance of scribble labels, employs cross pseudo-supervision and consistency regularization to improve the performance of the segmentation network. Furthermore, we introduce the random patch-based self-training framework, aimed at alleviating the performance constraints encountered in IA segmentation due to the limited availability of annotated DSA data. Our extensive experiments on the DIAS dataset demonstrate the effectiveness of these methods as potential baselines for future research and clinical applications. The dataset and code are publicly available at https://doi.org/10.5281/zenodo.11401368 and https://github.com/lseventeen/DIAS.

A Hybrid System for Automatic Identification of Corneal Layers on In Vivo Confocal Microscopy Images.

Purpose: Accurate identification of corneal layers with in vivo confocal microscopy (IVCM) is essential for the correct assessment of corneal lesions. This project aims to obtain a reliable automated identification of corneal layers from IVCM images. Methods: A total of 7957 IVCM images were included for model training and testing. Scanning depth information and pixel information of IVCM images were used to build the classification system. Firstly, two base classifiers based on convolutional neural networks and K-nearest neighbors were constructed. Second, two hybrid strategies, namely weighted voting method and light gradient boosting machine (LightGBM) algorithm were used to fuse the results from the two base classifiers and obtain the final classification. Finally, the confidence of prediction results was stratified to help find out model errors. Results: Both two hybrid systems outperformed the two base classifiers. The weighted area under the curve, weighted precision, weighted recall, and weighted F1 score were 0.9841, 0.9096, 0.9145, and 0.9111 for weighted voting hybrid system, and were 0.9794, 0.9039, 0.9055, and 0.9034 for the light gradient boosting machine stacking hybrid system, respectively. More than one-half of the misclassified samples were found using the confidence stratification method. Conclusions: The proposed hybrid approach could effectively integrate the scanning depth and pixel information of IVCM images, allowing for the accurate identification of corneal layers for grossly normal IVCM images. The confidence stratification approach was useful to find out misclassification of the system. Translational Relevance: The proposed hybrid approach lays important groundwork for the automatic identification of the corneal layer for IVCM images.

CKD-TransBTS: Clinical Knowledge-Driven Hybrid Transformer With Modality-Correlated Cross-Attention for Brain Tumor Segmentation.

Brain tumor segmentation (BTS) in magnetic resonance image (MRI) is crucial for brain tumor diagnosis, cancer management and research purposes. With the great success of the ten-year BraTS challenges as well as the advances of CNN and Transformer algorithms, a lot of outstanding BTS models have been proposed to tackle the difficulties of BTS in different technical aspects. However, existing studies hardly consider how to fuse the multi-modality images in a reasonable manner. In this paper, we leverage the clinical knowledge of how radiologists diagnose brain tumors from multiple MRI modalities and propose a clinical knowledge-driven brain tumor segmentation model, called CKD-TransBTS. Instead of directly concatenating all the modalities, we re-organize the input modalities by separating them into two groups according to the imaging principle of MRI. A dual-branch hybrid encoder with the proposed modality-correlated cross-attention block (MCCA) is designed to extract the multi-modality image features. The proposed model inherits the strengths from both Transformer and CNN with the local feature representation ability for precise lesion boundaries and long-range feature extraction for 3D volumetric images. To bridge the gap between Transformer and CNN features, we propose a Trans&CNN Feature Calibration block (TCFC) in the decoder. We compare the proposed model with six CNN-based models and six transformer-based models on the BraTS 2021 challenge dataset. Extensive experiments demonstrate that the proposed model achieves state-of-the-art brain tumor segmentation performance compared with all the competitors.

HoVer-Trans: Anatomy-Aware HoVer-Transformer for ROI-Free Breast Cancer Diagnosis in Ultrasound Images.

Ultrasonography is an important routine examination for breast cancer diagnosis, due to its non-invasive, radiation-free and low-cost properties. However, the diagnostic accuracy of breast cancer is still limited due to its inherent limitations. Then, a precise diagnose using breast ultrasound (BUS) image would be significant useful. Many learning-based computer-aided diagnostic methods have been proposed to achieve breast cancer diagnosis/lesion classification. However, most of them require a pre-define region of interest (ROI) and then classify the lesion inside the ROI. Conventional classification backbones, such as VGG16 and ResNet50, can achieve promising classification results with no ROI requirement. But these models lack interpretability, thus restricting their use in clinical practice. In this study, we propose a novel ROI-free model for breast cancer diagnosis in ultrasound images with interpretable feature representations. We leverage the anatomical prior knowledge that malignant and benign tumors have different spatial relationships between different tissue layers, and propose a HoVer-Transformer to formulate this prior knowledge. The proposed HoVer-Trans block extracts the inter- and intra-layer spatial information horizontally and vertically. We conduct and release an open dataset GDPH&SYSUCC for breast cancer diagnosis in BUS. The proposed model is evaluated in three datasets by comparing with four CNN-based models and three vision transformer models via five-fold cross validation. It achieves state-of-the-art classification performance (GDPH&SYSUCC AUC: 0.924, ACC: 0.893, Spec: 0.836, Sens: 0.926) with the best model interpretability. In the meanwhile, our proposed model outperforms two senior sonographers on the breast cancer diagnosis when only one BUS image is given (GDPH&SYSUCC-AUC ours: 0.924 vs. reader1: 0.825 vs. reader2: 0.820).

Computerized tertiary lymphoid structures density on H&E-images is a prognostic biomarker in resectable lung adenocarcinoma.

The increased amount of tertiary lymphoid structures (TLSs) is associated with a favorable prognosis in patients with lung adenocarcinoma (LUAD). However, evaluating TLSs manually is an experience-dependent and time-consuming process, which limits its clinical application. In this multi-center study, we developed an automated computational workflow for quantifying the TLS density in the tumor region of routine hematoxylin and eosin (H&E)-stained whole-slide images (WSIs). The association between the computerized TLS density and disease-free survival (DFS) was further explored in 802 patients with resectable LUAD of three cohorts. Additionally, a Cox proportional hazard regression model, incorporating clinicopathological variables and the TLS density, was established to assess its prognostic ability. The computerized TLS density was an independent prognostic biomarker in patients with resectable LUAD. The integration of the TLS density with clinicopathological variables could support individualized clinical decision-making by improving prognostic stratification.

SMILE: Cost-sensitive multi-task learning for nuclear segmentation and classification with imbalanced annotations.

High throughput nuclear segmentation and classification of whole slide images (WSIs) is crucial to biological analysis, clinical diagnosis and precision medicine. With the advances of CNN algorithms and the continuously growing datasets, considerable progress has been made in nuclear segmentation and classification. However, few works consider how to reasonably deal with nuclear heterogeneity in the following two aspects: imbalanced data distribution and diversified morphology characteristics. The minority classes might be dominated by the majority classes due to the imbalanced data distribution and the diversified morphology characteristics may lead to fragile segmentation results. In this study, a cost-Sensitive MultI-task LEarning (SMILE) framework is conducted to tackle the data heterogeneity problem. Based on the most popular multi-task learning backbone in nuclei segmentation and classification, we propose a multi-task correlation attention (MTCA) to perform feature interaction of multiple high relevant tasks to learn better feature representation. A cost-sensitive learning strategy is proposed to solve the imbalanced data distribution by increasing the penalization for the error classification of the minority classes. Furthermore, we propose a novel post-processing step based on the coarse-to-fine marker-controlled watershed scheme to alleviate fragile segmentation when nuclei are with large size and unclear contour. Extensive experiments show that the proposed method achieves state-of-the-art performances on CoNSeP and MoNuSAC 2020 datasets. The code is available at: https://github.com/panxipeng/nuclear_segandcls.

Artificial intelligence-quantified tumour-lymphocyte spatial interaction predicts disease-free survival in resected lung adenocarcinoma: A graph-based, multicentre study.

BACKGROUND AND OBJECTIVE: A high degree of lymphocyte infiltration is related to superior outcomes amongst patients with lung adenocarcinoma. Recent evidence indicates that the spatial interactions between tumours and lymphocytes also influence the anti-tumour immune responses, but the spatial analysis at the cellular level remains insufficient. METHODS: We proposed an artificial intelligence-quantified Tumour-Lymphocyte Spatial Interaction score (TLSI-score) by calculating the ratio between the number of spatial adjacent tumour-lymphocyte and the number of tumour cells based on topology cell graph constructed using H&E-stained whole-slide images. The association of TLSI-score with disease-free survival (DFS) was explored in 529 patients with lung adenocarcinoma across three independent cohorts (D1, 275; V1, 139; V2, 115). RESULTS: After adjusting for pTNM stage and other clinicopathologic risk factors, a higher TLSI-score was independently associated with longer DFS than a low TLSI-score in the three cohorts [D1, adjusted hazard ratio (HR), 0.674; 95% confidence interval (CI) 0.463-0.983; p = 0.040; V1, adjusted HR, 0.408; 95% CI 0.223-0.746; p = 0.004; V2, adjusted HR, 0.294; 95% CI 0.130-0.666; p = 0.003]. By integrating the TLSI-score with clinicopathologic risk factors, the integrated model (full model) improves the prediction of DFS in three independent cohorts (C-index, D1, 0.716 vs. 0.701; V1, 0.666 vs. 0.645; V2, 0.708 vs. 0.662) CONCLUSIONS: TLSI-score shows the second highest relative contribution to the prognostic prediction model, next to the pTNM stage. TLSI-score can assist in the characterising of tumour microenvironment and is expected to promote individualized treatment and follow-up decision-making in clinical practice.