Federated Learning for Decentralized Artificial Intelligence in Melanoma Diagnostics.

Importance: The development of artificial intelligence (AI)-based melanoma classifiers typically calls for large, centralized datasets, requiring hospitals to give away their patient data, which raises serious privacy concerns. To address this concern, decentralized federated learning has been proposed, where classifier development is distributed across hospitals. Objective: To investigate whether a more privacy-preserving federated learning approach can achieve comparable diagnostic performance to a classical centralized (ie, single-model) and ensemble learning approach for AI-based melanoma diagnostics. Design, Setting, and Participants: This multicentric, single-arm diagnostic study developed a federated model for melanoma-nevus classification using histopathological whole-slide images prospectively acquired at 6 German university hospitals between April 2021 and February 2023 and benchmarked it using both a holdout and an external test dataset. Data analysis was performed from February to April 2023. Exposures: All whole-slide images were retrospectively analyzed by an AI-based classifier without influencing routine clinical care. Main Outcomes and Measures: The area under the receiver operating characteristic curve (AUROC) served as the primary end point for evaluating the diagnostic performance. Secondary end points included balanced accuracy, sensitivity, and specificity. Results: The study included 1025 whole-slide images of clinically melanoma-suspicious skin lesions from 923 patients, consisting of 388 histopathologically confirmed invasive melanomas and 637 nevi. The median (range) age at diagnosis was 58 (18-95) years for the training set, 57 (18-93) years for the holdout test dataset, and 61 (18-95) years for the external test dataset; the median (range) Breslow thickness was 0.70 (0.10-34.00) mm, 0.70 (0.20-14.40) mm, and 0.80 (0.30-20.00) mm, respectively. The federated approach (0.8579; 95% CI, 0.7693-0.9299) performed significantly worse than the classical centralized approach (0.9024; 95% CI, 0.8379-0.9565) in terms of AUROC on a holdout test dataset (pairwise Wilcoxon signed-rank, P < .001) but performed significantly better (0.9126; 95% CI, 0.8810-0.9412) than the classical centralized approach (0.9045; 95% CI, 0.8701-0.9331) on an external test dataset (pairwise Wilcoxon signed-rank, P < .001). Notably, the federated approach performed significantly worse than the ensemble approach on both the holdout (0.8867; 95% CI, 0.8103-0.9481) and external test dataset (0.9227; 95% CI, 0.8941-0.9479). Conclusions and Relevance: The findings of this diagnostic study suggest that federated learning is a viable approach for the binary classification of invasive melanomas and nevi on a clinically representative distributed dataset. Federated learning can improve privacy protection in AI-based melanoma diagnostics while simultaneously promoting collaboration across institutions and countries. Moreover, it may have the potential to be extended to other image classification tasks in digital cancer histopathology and beyond.

CpG-biomarkers in tumor tissue and prediction models for the survival of colorectal cancer: A systematic review and external validation study.

The research aimed to identify previously published CpG-methylation-based prognostic biomarkers and prediction models for colorectal cancer (CRC) prognosis and validate them in a large external cohort. A systematic search was conducted, analyzing 298 unique CpGs and 12 CpG-based prognostic models from 28 studies. After adjustment for clinical variables, 48 CpGs and five prognostic models were confirmed to be associated with survival. However, the discrimination ability of the models was insufficient, with area under the receiver operating characteristic curves ranging from 0.53 to 0.62. Calibration accuracy was mostly poor, and no significant added prognostic value beyond traditional clinical variables was observed. All prognostic models were rated at high risk of bias. While a fraction of CpGs showed potential clinical utility and generalizability, the CpG-based prognostic models performed poorly and lacked clinical relevance.

Novel biomarker SARIFA in colorectal cancer: highly prognostic, not genetically driven and histologic indicator of a distinct tumor biology.

SARIFA (Stroma AReactive Invasion Front Areas) has recently emerged as a promising histopathological biomarker for colon and gastric cancer. To elucidate the underlying tumor biology, we assessed SARIFA-status in tissue specimens from The-Cancer-Genome-Atlas (TCGA) cohorts COAD (colonic adenocarcinoma) and READ (rectal adenocarcinoma). For the final analysis, 207 CRC patients could be included, consisting of 69 SARIFA-positive and 138 SARIFA-negative cases. In this external validation cohort, H&E-based SARIFA-positivity was strongly correlated with unfavorable overall, disease-specific, and progression-free survival, partly outperforming conventional prognostic factors. SARIFA-positivity was not associated with known high-risk genetic profiles, such as BRAF V600E mutations or microsatellite-stable status. Transcriptionally, SARIFA-positive CRCs exhibited an overlap with CRC consensus molecular subtypes CMS1 and CMS4, along with distinct differential gene expression patterns, linked to lipid metabolism and increased stromal cell infiltration scores (SIIS). Gene-expression-based drug sensitivity prediction revealed a differential treatment response in SARIFA-positive CRCs. In conclusion, SARIFA represents the H&E-based counterpart of an aggressive tumor biology, demonstrating a partial overlap with CMS1/4 and also adding a further biological layer related to lipid metabolism. Our findings underscore SARIFA-status as an ideal biomarker for refined patient stratification and novel drug developments, particularly given its cost-effective assessment based on routinely available H&E slides.

Deep learning trained on lymph node status predicts outcome from gastric cancer histopathology: a retrospective multicentric study.

AIM: Gastric cancer (GC) is a tumour entity with highly variant outcomes. Lymph node metastasis is a prognostically adverse biomarker. We hypothesised that GC primary tissue contains information that is predictive of lymph node status and patient prognosis and that this information can be extracted using deep learning (DL). METHODS: Using three patient cohorts comprising 1146 patients, we trained and validated a DL system to predict lymph node status directly from haematoxylin and eosin-stained GC tissue sections. We investigated the concordance between the DL-based prediction from the primary tumour slides (aiN score) and the histopathological lymph node status (pN). Furthermore, we assessed the prognostic value of the aiN score alone and when combined with the pN status. RESULTS: The aiN score predicted the pN status reaching area under the receiver operating characteristic curves of 0.71 in the training cohort and 0.69 and 0.65 in the two test cohorts. In a multivariate Cox analysis, the aiN score was an independent predictor of patient survival with hazard ratios of 1.5 in the training cohort and of 1.3 and 2.2 in the two test cohorts. A combination of the aiN score and the pN status prognostically stratified patients by survival with p-values <0.05 in logrank tests. CONCLUSION: GC primary tumour tissue contains additional prognostic information that is accessible using the aiN score. In combination with the pN status, this can be used for personalised management of GC patients after prospective validation.

Sublethal necroptosis signaling promotes inflammation and liver cancer.

It is currently not well known how necroptosis and necroptosis responses manifest in vivo. Here, we uncovered a molecular switch facilitating reprogramming between two alternative modes of necroptosis signaling in hepatocytes, fundamentally affecting immune responses and hepatocarcinogenesis. Concomitant necrosome and NF-κB activation in hepatocytes, which physiologically express low concentrations of receptor-interacting kinase 3 (RIPK3), did not lead to immediate cell death but forced them into a prolonged "sublethal" state with leaky membranes, functioning as secretory cells that released specific chemokines including CCL20 and MCP-1. This triggered hepatic cell proliferation as well as activation of procarcinogenic monocyte-derived macrophage cell clusters, contributing to hepatocarcinogenesis. In contrast, necrosome activation in hepatocytes with inactive NF-κB-signaling caused an accelerated execution of necroptosis, limiting alarmin release, and thereby preventing inflammation and hepatocarcinogenesis. Consistently, intratumoral NF-κB-necroptosis signatures were associated with poor prognosis in human hepatocarcinogenesis. Therefore, pharmacological reprogramming between these distinct forms of necroptosis may represent a promising strategy against hepatocellular carcinoma.

Validation of the prognostic value of CD3 and CD8 cell densities analogous to the Immunoscore® by stage and location of colorectal cancer: an independent patient cohort study.

In addition to the traditional staging system in colorectal cancer (CRC), the Immunoscore® has been proposed to characterize the level of immune infiltration in tumor tissue and as a potential prognostic marker. The aim of this study was to examine and validate associations of an immune cell score analogous to the Immunoscore® with established molecular tumor markers and with CRC patient survival in a routine setting. Patients from a population-based cohort study with available CRC tumor tissue blocks were included in this analysis. CD3+ and CD8+ tumor infiltrating lymphocytes in the tumor center and invasive margin were determined in stained tumor tissue slides. Based on the T-cell density in each region, an  immune cell score closely analogous to the concept of the Immunoscore® was calculated and tumors categorized into IS-low, IS-intermediate, or IS-high. Logistic regression models were used to assess associations between clinicopathological characteristics with the immune cell score, and Cox proportional hazards models to analyze associations with cancer-specific, relapse-free, and overall survival. From 1,535 patients with CRC, 411 (27%) had IS-high tumors. Microsatellite instability (MSI-high) was strongly associated with higher immune cell score levels (p < 0.001). Stage I-III patients with IS-high had better CRC-specific and relapse-free survival compared to patients with IS-low (hazard ratio [HR] = 0.42 [0.27-0.66] and HR = 0.45 [0.31-0.67], respectively). Patients with microsatellite stable (MSS) tumors and IS-high had better survival (HRCSS  = 0.60 [0.42-0.88]) compared to MSS/IS-low patients. In this population-based cohort of CRC patients, the immune cell score was significantly associated with better patient survival. It was a similarly strong prognostic marker in patients with MSI-high tumors and in the larger group of patients with MSS tumors. Additionally, this study showed that it is possible to implement an analogous immune cell score approach and validate the Immunoscore® using open source software in an academic setting. Thus, the Immunoscore® could be useful to improve the traditional staging system in colon and rectal cancer used in clinical practice.

Second-line therapy with nivolumab plus ipilimumab for older patients with oesophageal squamous cell cancer (RAMONA): a multicentre, open-label phase 2 trial.

BACKGROUND: The overall survival of patients with advanced and refractory oesophageal squamous cell carcinoma, mostly aged 65 years and older, is poor. Treatment with PD-1 antibodies showed improved progression-free survival and overall survival. We assessed the safety and efficacy of combined nivolumab and ipilimumab therapy in this population. METHODS: This multicentre, open-label, phase 2 trial done in 32 sites in Germany included patients aged 65 years and older with oesophageal squamous cell carcinoma and disease progression or recurrence following first-line therapy. Patients were treated with nivolumab (240 mg fixed dose once every 2 weeks, intravenously) in the safety run-in phase and continued with nivolumab and ipilimumab (nivolumab 240 mg fixed dose once every 2 weeks and ipilimumab 1 mg/kg once every 6 weeks, intravenously). The primary endpoint was overall survival, which was compared with a historical cohort receiving standard chemotherapy in the intention-to-treat population. This study is registered with ClinicalTrials.gov, NCT03416244. FINDINGS: Between March 2, 2018, and Aug 20, 2020, we screened 75 patients with advanced oesophageal squamous cell carcinoma. We enrolled 66 patients (50 [76%] men and 16 [24%] women; median age 70·5 years [IQR 67·0-76·0]), 44 (67%) of whom received combined nivolumab and ipilimumab therapy and 22 (33%) received nivolumab alone. Median overall survival time at the prespecified data cutoff was 7·2 months (95% CI 5·7-12·4) and significantly higher than in a historical cohort receiving standard chemotherapy (p=0·0063). The most common treatment-related adverse events were fatigue (12 [29%] of 42), nausea (11 [26%]), and diarrhoea (ten [24%]). Grade 3-5 treatment-related adverse events occurred in 13 (20%) of 66 patients. Treatment-related death occurred in one patient with bronchiolitis obliterans while on nivolumab and ipilimumab treatment. INTERPRETATION: Patients aged at least 65 years, with advanced oesophageal squamous cell carcinoma might benefit from combined nivolumab and ipilimumab therapy in second-line treatment. FUNDING: Bristol Myers Squibb.

Golgi Protein 73 (GP73) Serum Levels Predict Outcome after Resection of Biliary Tract Cancer.

BACKGROUND: Tumor resection represents the only potentially curative therapy for patients with biliary tract cancer. Nevertheless, disease recurrence is observed in about 50% of patients, leading to a 5-years survival rate of less than 50%. The Golgi protein 73 (GP73), a type II Golgi transmembrane protein, exerts important functions of intracellular protein processing and transportation. Circulating GP73 has recently been suggested as a prognostic marker following resection of hepatocellular carcinoma (HCC) but its role in the context of BTC has remained unknown. In this study, we evaluate a potential role of circulating GP73 as a novel biomarker in patients with resectable BTC. METHODS: GP73 serum levels were measured by immunoassay in n = 97 BTC and n = 40 HCC patients as well as n = 31 healthy controls. Results were correlated with clinical data. RESULTS: Serum GP73 levels were significantly elevated in BTC patients compared to healthy controls but lower compared to HCC patients. The combination of GP73/CA19-9 showed a sensitivity and specificity of 83.5% and 90.3% regarding the differentiation of BTC patients and healthy controls. BTC patients with baseline GP73 levels above the ideal cut-off value (42.47 ng/mL) showed a significantly reduced median overall survival (193 days) compared to patients with preoperative GP73 levels below this cut-off (882 days). These results were confirmed in uni- and multivariate Cox-regression analysis including several clinicopathological parameters such as age, ECOG performance status, tumor stage as well as established tumor markers and parameters of liver and kidney function. CONCLUSIONS: GP73 represents a previously unrecognized biomarker in the patients with resectable BTC that identifies patients with an impaired postoperative outcome. If larger clinical trials confirmed these findings, measurement of GP73 serum levels might become a novel tool in the challenging preoperative stratification process of patients with resectable BTC.

Model soups improve performance of dermoscopic skin cancer classifiers.

BACKGROUND: Image-based cancer classifiers suffer from a variety of problems which negatively affect their performance. For example, variation in image brightness or different cameras can already suffice to diminish performance. Ensemble solutions, where multiple model predictions are combined into one, can improve these problems. However, ensembles are computationally intensive and less transparent to practitioners than single model solutions. Constructing model soups, by averaging the weights of multiple models into a single model, could circumvent these limitations while still improving performance. OBJECTIVE: To investigate the performance of model soups for a dermoscopic melanoma-nevus skin cancer classification task with respect to (1) generalisation to images from other clinics, (2) robustness against small image changes and (3) calibration such that the confidences correspond closely to the actual predictive uncertainties. METHODS: We construct model soups by fine-tuning pre-trained models on seven different image resolutions and subsequently averaging their weights. Performance is evaluated on a multi-source dataset including holdout and external components. RESULTS: We find that model soups improve generalisation and calibration on the external component while maintaining performance on the holdout component. For robustness, we observe performance improvements for pertubated test images, while the performance on corrupted test images remains on par. CONCLUSIONS: Overall, souping for skin cancer classifiers has a positive effect on generalisation, robustness and calibration. It is easy for practitioners to implement and by combining multiple models into a single model, complexity is reduced. This could be an important factor in achieving clinical applicability, as less complexity generally means more transparency.

Deep learning can predict survival directly from histology in clear cell renal cell carcinoma.

For clear cell renal cell carcinoma (ccRCC) risk-dependent diagnostic and therapeutic algorithms are routinely implemented in clinical practice. Artificial intelligence-based image analysis has the potential to improve outcome prediction and thereby risk stratification. Thus, we investigated whether a convolutional neural network (CNN) can extract relevant image features from a representative hematoxylin and eosin-stained slide to predict 5-year overall survival (5y-OS) in ccRCC. The CNN was trained to predict 5y-OS in a binary manner using slides from TCGA and validated using an independent in-house cohort. Multivariable logistic regression was used to combine of the CNNs prediction and clinicopathological parameters. A mean balanced accuracy of 72.0% (standard deviation [SD] = 7.9%), sensitivity of 72.4% (SD = 10.6%), specificity of 71.7% (SD = 11.9%) and area under receiver operating characteristics curve (AUROC) of 0.75 (SD = 0.07) was achieved on the TCGA training set (n = 254 patients / WSIs) using 10-fold cross-validation. On the external validation cohort (n = 99 patients / WSIs), mean accuracy, sensitivity, specificity and AUROC were 65.5% (95%-confidence interval [CI]: 62.9-68.1%), 86.2% (95%-CI: 81.8-90.5%), 44.9% (95%-CI: 40.2-49.6%), and 0.70 (95%-CI: 0.69-0.71). A multivariable model including age, tumor stage and metastasis yielded an AUROC of 0.75 on the TCGA cohort. The inclusion of the CNN-based classification (Odds ratio = 4.86, 95%-CI: 2.70-8.75, p < 0.01) raised the AUROC to 0.81. On the validation cohort, both models showed an AUROC of 0.88. In univariable Cox regression, the CNN showed a hazard ratio of 3.69 (95%-CI: 2.60-5.23, p < 0.01) on TCGA and 2.13 (95%-CI: 0.92-4.94, p = 0.08) on external validation. The results demonstrate that the CNN's image-based prediction of survival is promising and thus this widely applicable technique should be further investigated with the aim of improving existing risk stratification in ccRCC.

Clinical relevance of biomarkers in cholangiocarcinoma: critical revision and future directions.

Cholangiocarcinoma (CCA) is a malignant tumour arising from the biliary system. In Europe, this tumour frequently presents as a sporadic cancer in patients without defined risk factors and is usually diagnosed at advanced stages with a consequent poor prognosis. Therefore, the identification of biomarkers represents an utmost need for patients with CCA. Numerous studies proposed a wide spectrum of biomarkers at tissue and molecular levels. With the present paper, a multidisciplinary group of experts within the European Network for the Study of Cholangiocarcinoma discusses the clinical role of tissue biomarkers and provides a selection based on their current relevance and potential applications in the framework of CCA. Recent advances are proposed by dividing biomarkers based on their potential role in diagnosis, prognosis and therapy response. Limitations of current biomarkers are also identified, together with specific promising areas (ie, artificial intelligence, patient-derived organoids, targeted therapy) where research should be focused to develop future biomarkers.

Explainable artificial intelligence in skin cancer recognition: A systematic review.

BACKGROUND: Due to their ability to solve complex problems, deep neural networks (DNNs) are becoming increasingly popular in medical applications. However, decision-making by such algorithms is essentially a black-box process that renders it difficult for physicians to judge whether the decisions are reliable. The use of explainable artificial intelligence (XAI) is often suggested as a solution to this problem. We investigate how XAI is used for skin cancer detection: how is it used during the development of new DNNs? What kinds of visualisations are commonly used? Are there systematic evaluations of XAI with dermatologists or dermatopathologists? METHODS: Google Scholar, PubMed, IEEE Explore, Science Direct and Scopus were searched for peer-reviewed studies published between January 2017 and October 2021 applying XAI to dermatological images: the search terms histopathological image, whole-slide image, clinical image, dermoscopic image, skin, dermatology, explainable, interpretable and XAI were used in various combinations. Only studies concerned with skin cancer were included. RESULTS: 37 publications fulfilled our inclusion criteria. Most studies (19/37) simply applied existing XAI methods to their classifier to interpret its decision-making. Some studies (4/37) proposed new XAI methods or improved upon existing techniques. 14/37 studies addressed specific questions such as bias detection and impact of XAI on man-machine-interactions. However, only three of them evaluated the performance and confidence of humans using CAD systems with XAI. CONCLUSION: XAI is commonly applied during the development of DNNs for skin cancer detection. However, a systematic and rigorous evaluation of its usefulness in this scenario is lacking.

Artificial Intelligence-based Detection of FGFR3 Mutational Status Directly from Routine Histology in Bladder Cancer: A Possible Preselection for Molecular Testing?

BACKGROUND: Fibroblast growth factor receptor (FGFR) inhibitor treatment has become the first clinically approved targeted therapy in bladder cancer. However, it requires previous molecular testing of each patient, which is costly and not ubiquitously available. OBJECTIVE: To determine whether an artificial intelligence system is able to predict mutations of the FGFR3 gene directly from routine histology slides of bladder cancer. DESIGN, SETTING, AND PARTICIPANTS: We trained a deep learning network to detect FGFR3 mutations on digitized slides of muscle-invasive bladder cancers stained with hematoxylin and eosin from the Cancer Genome Atlas (TCGA) cohort (n = 327) and validated the algorithm on the "Aachen" cohort (n = 182; n = 121 pT2-4, n = 34 stroma-invasive pT1, and n = 27 noninvasive pTa tumors). OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS: The primary endpoint was the area under the receiver operating curve (AUROC) for mutation detection. Performance of the deep learning system was compared with visual scoring by an uropathologist. RESULTS AND LIMITATIONS: In the TCGA cohort, FGFR3 mutations were detected with an AUROC of 0.701 (p < 0.0001). In the Aachen cohort, FGFR3 mutants were found with an AUROC of 0.725 (p < 0.0001). When trained on TCGA, the network generalized to the Aachen cohort, and detected FGFR3 mutants with an AUROC of 0.625 (p = 0.0112). A subgroup analysis and histological evaluation found highest accuracy in papillary growth, luminal gene expression subtypes, females, and American Joint Committee on Cancer (AJCC) stage II tumors. In a head-to-head comparison, the deep learning system outperformed the uropathologist in detecting FGFR3 mutants. CONCLUSIONS: Our computer-based artificial intelligence system was able to detect genetic alterations of the FGFR3 gene of bladder cancer patients directly from histological slides. In the future, this system could be used to preselect patients for further molecular testing. However, analyses of larger, multicenter, muscle-invasive bladder cancer cohorts are now needed in order to validate and extend our findings. PATIENT SUMMARY: In this report, a computer-based artificial intelligence (AI) system was applied to histological slides to predict genetic alterations of the FGFR3 gene in bladder cancer. We found that the AI system was able to find the alteration with high accuracy. In the future, this system could be used to preselect patients for further molecular testing.

Deep learning-based classification of kidney transplant pathology: a retrospective, multicentre, proof-of-concept study.

BACKGROUND: Histopathological assessment of transplant biopsies is currently the standard method to diagnose allograft rejection and can help guide patient management, but it is one of the most challenging areas of pathology, requiring considerable expertise, time, and effort. We aimed to analyse the utility of deep learning to preclassify histology of kidney allograft biopsies into three main broad categories (ie, normal, rejection, and other diseases) as a potential biopsy triage system focusing on transplant rejection. METHODS: We performed a retrospective, multicentre, proof-of-concept study using 5844 digital whole slide images of kidney allograft biopsies from 1948 patients. Kidney allograft biopsy samples were identified by a database search in the Departments of Pathology of the Amsterdam UMC, Amsterdam, Netherlands (1130 patients) and the University Medical Center Utrecht, Utrecht, Netherlands (717 patients). 101 consecutive kidney transplant biopsies were identified in the archive of the Institute of Pathology, RWTH Aachen University Hospital, Aachen, Germany. Convolutional neural networks (CNNs) were trained to classify allograft biopsies as normal, rejection, or other diseases. Three times cross-validation (1847 patients) and deployment on an external real-world cohort (101 patients) were used for validation. Area under the receiver operating characteristic curve (AUROC) was used as the main performance metric (the primary endpoint to assess CNN performance). FINDINGS: Serial CNNs, first classifying kidney allograft biopsies as normal (AUROC 0·87 [ten times bootstrapped CI 0·85-0·88]) and disease (0·87 [0·86-0·88]), followed by a second CNN classifying biopsies classified as disease into rejection (0·75 [0·73-0·76]) and other diseases (0·75 [0·72-0·77]), showed similar AUROC in cross-validation and deployment on independent real-world data (first CNN normal AUROC 0·83 [0·80-0·85], disease 0·83 [0·73-0·91]; second CNN rejection 0·61 [0·51-0·70], other diseases 0·61 [0·50-0·74]). A single CNN classifying biopsies as normal, rejection, or other diseases showed similar performance in cross-validation (normal AUROC 0·80 [0·73-0·84], rejection 0·76 [0·66-0·80], other diseases 0·50 [0·36-0·57]) and generalised well for normal and rejection classes in the real-world data. Visualisation techniques highlighted rejection-relevant areas of biopsies in the tubulointerstitium. INTERPRETATION: This study showed that deep learning-based classification of transplant biopsies could support pathological diagnostics of kidney allograft rejection. FUNDING: European Research Council; German Research Foundation; German Federal Ministries of Education and Research, Health, and Economic Affairs and Energy; Dutch Kidney Foundation; Human(e) AI Research Priority Area of the University of Amsterdam; and Max-Eder Programme of German Cancer Aid.

Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review.

BACKGROUND: Over the past decade, the development of molecular high-throughput methods (omics) increased rapidly and provided new insights for cancer research. In parallel, deep learning approaches revealed the enormous potential for medical image analysis, especially in digital pathology. Combining image and omics data with deep learning tools may enable the discovery of new cancer biomarkers and a more precise prediction of patient prognosis. This systematic review addresses different multimodal fusion methods of convolutional neural network-based image analyses with omics data, focussing on the impact of data combination on the classification performance. METHODS: PubMed was screened for peer-reviewed articles published in English between January 2015 and June 2021 by two independent researchers. Search terms related to deep learning, digital pathology, omics, and multimodal fusion were combined. RESULTS: We identified a total of 11 studies meeting the inclusion criteria, namely studies that used convolutional neural networks for haematoxylin and eosin image analysis of patients with cancer in combination with integrated omics data. Publications were categorised according to their endpoints: 7 studies focused on survival analysis and 4 studies on prediction of cancer subtypes, malignancy or microsatellite instability with spatial analysis. CONCLUSIONS: Image-based classifiers already show high performances in prognostic and predictive cancer diagnostics. The integration of omics data led to improved performance in all studies described here. However, these are very early studies that still require external validation to demonstrate their generalisability and robustness. Further and more comprehensive studies with larger sample sizes are needed to evaluate performance and determine clinical benefits.

Deep learning can predict lymph node status directly from histology in colorectal cancer.

BACKGROUND: Lymph node status is a prognostic marker and strongly influences therapeutic decisions in colorectal cancer (CRC). OBJECTIVES: The objective of the study is to investigate whether image features extracted by a deep learning model from routine histological slides and/or clinical data can be used to predict CRC lymph node metastasis (LNM). METHODS: Using histological whole slide images (WSIs) of primary tumours of 2431 patients in the DACHS cohort, we trained a convolutional neural network to predict LNM. In parallel, we used clinical data derived from the same cases in logistic regression analyses. Subsequently, the slide-based artificial intelligence predictor (SBAIP) score was included in the regression. WSIs and data from 582 patients of the TCGA cohort were used as the external test set. RESULTS: On the internal test set, the SBAIP achieved an area under receiver operating characteristic (AUROC) of 71.0%, the clinical classifier achieved an AUROC of 67.0% and a combination of the two classifiers yielded an improvement to 74.1%. Whereas the clinical classifier's performance remained stable on the TCGA set, performance of the SBAIP dropped to an AUROC of 61.2%. Performance of the clinical classifier depended strongly on the T stage. CONCLUSION: Deep learning-based image analysis may help predict LNM of patients with CRC using routine histological slides. Combination with clinical data such as T stage might be useful. Strategies to increase performance of the SBAIP on external images should be investigated.