Federated 3D multi-organ segmentation with partially labeled and unlabeled data.

PURPOSE: This paper considers a new problem setting for multi-organ segmentation based on the following observations. In reality, (1) collecting a large-scale dataset from various institutes is usually impeded due to privacy issues; (2) many images are not labeled since the slice-by-slice annotation is costly; and (3) datasets may exhibit inconsistent, partial annotations across different institutes. Learning a federated model from these distributed, partially labeled, and unlabeled samples is an unexplored problem. METHODS: To simulate this multi-organ segmentation problem, several distributed clients and a central server are maintained. The central server coordinates with clients to learn a global model using distributed private datasets, which comprise a small part of partially labeled images and a large part of unlabeled images. To address this problem, a practical framework that unifies partially supervised learning (PSL), semi-supervised learning (SSL), and federated learning (FL) paradigms with PSL, SSL, and FL modules is proposed. The PSL module manages to learn from partially labeled samples. The SSL module extracts valuable information from unlabeled data. Besides, the FL module aggregates local information from distributed clients to generate a global statistical model. With the collaboration of three modules, the presented scheme could take advantage of these distributed imperfect datasets to train a generalizable model. RESULTS: The proposed method was extensively evaluated with multiple abdominal CT datasets, achieving an average result of 84.83% in Dice and 41.62 mm in 95HD for multi-organ (liver, spleen, and stomach) segmentation. Moreover, its efficacy in transfer learning further demonstrated its good generalization ability for downstream segmentation tasks. CONCLUSION: This study considers a novel problem of multi-organ segmentation, which aims to develop a generalizable model using distributed, partially labeled, and unlabeled CT images. A practical framework is presented, which, through extensive validation, has proved to be an effective solution, demonstrating strong potential in addressing this challenging problem.

Towards better laparoscopic video segmentation: A class-wise contrastive learning approach with multi-scale feature extraction.

The task of segmentation is integral to computer-aided surgery systems. Given the privacy concerns associated with medical data, collecting a large amount of annotated data for training is challenging. Unsupervised learning techniques, such as contrastive learning, have shown powerful capabilities in learning image-level representations from unlabelled data. This study leverages classification labels to enhance the accuracy of the segmentation model trained on limited annotated data. The method uses a multi-scale projection head to extract image features at various scales. The partitioning method for positive sample pairs is then improved to perform contrastive learning on the extracted features at each scale to effectively represent the differences between positive and negative samples in contrastive learning. Furthermore, the model is trained simultaneously with both segmentation labels and classification labels. This enables the model to extract features more effectively from each segmentation target class and further accelerates the convergence speed. The method was validated using the publicly available CholecSeg8k dataset for comprehensive abdominal cavity surgical segmentation. Compared to select existing methods, the proposed approach significantly enhances segmentation performance, even with a small labelled subset (1-10%) of the dataset, showcasing a superior intersection over union (IoU) score.

Revisiting instrument segmentation: Learning from decentralized surgical sequences with various imperfect annotations.

This paper focuses on a new and challenging problem related to instrument segmentation. This paper aims to learn a generalizable model from distributed datasets with various imperfect annotations. Collecting a large-scale dataset for centralized learning is usually impeded due to data silos and privacy issues. Besides, local clients, such as hospitals or medical institutes, may hold datasets with diverse and imperfect annotations. These datasets can include scarce annotations (many samples are unlabelled), noisy labels prone to errors, and scribble annotations with less precision. Federated learning (FL) has emerged as an attractive paradigm for developing global models with these locally distributed datasets. However, its potential in instrument segmentation has yet to be fully investigated. Moreover, the problem of learning from various imperfect annotations in an FL setup is rarely studied, even though it presents a more practical and beneficial scenario. This work rethinks instrument segmentation in such a setting and propose a practical FL framework for this issue. Notably, this approach surpassed centralized learning under various imperfect annotation settings. This method established a foundational benchmark, and future work can build upon it by considering each client owning various annotations and aligning closer with real-world complexities.

YOLOv7-RepFPN: Improving real-time performance of laparoscopic tool detection on embedded systems.

This study focuses on enhancing the inference speed of laparoscopic tool detection on embedded devices. Laparoscopy, a minimally invasive surgery technique, markedly reduces patient recovery times and postoperative complications. Real-time laparoscopic tool detection helps assisting laparoscopy by providing information for surgical navigation, and its implementation on embedded devices is gaining interest due to the portability, network independence and scalability of the devices. However, embedded devices often face computation resource limitations, potentially hindering inference speed. To mitigate this concern, the work introduces a two-fold modification to the YOLOv7 model: the feature channels and integrate RepBlock is halved, yielding the YOLOv7-RepFPN model. This configuration leads to a significant reduction in computational complexity. Additionally, the focal EIoU (efficient intersection of union) loss function is employed for bounding box regression. Experimental results on an embedded device demonstrate that for frame-by-frame laparoscopic tool detection, the proposed YOLOv7-RepFPN achieved an mAP of 88.2% (with IoU set to 0.5) on a custom dataset based on EndoVis17, and an inference speed of 62.9 FPS. Contrasting with the original YOLOv7, which garnered an 89.3% mAP and 41.8 FPS under identical conditions, the methodology enhances the speed by 21.1 FPS while maintaining detection accuracy. This emphasizes the effectiveness of the work.

A clinical assessment of three-dimensional-printed liver model navigation for thrice or more repeated hepatectomy based on a conversation analysis.

PURPOSES: We performed a conversation analysis of the speech conducted among the surgical team during three-dimensional (3D)-printed liver model navigation for thrice or more repeated hepatectomy (TMRH). METHODS: Seventeen patients underwent 3D-printed liver navigation surgery for TMRH. After transcription of the utterances recorded during surgery, the transcribed utterances were coded by the utterer, utterance object, utterance content, sensor, and surgical process during conversation. We then analyzed the utterances and clarified the association between the surgical process and conversation through the intraoperative reference of the 3D-printed liver. RESULTS: In total, 130 conversations including 1648 segments were recorded. Utterance coding showed that the operator/assistant, 3D-printed liver/real liver, fact check (F)/plan check (Pc), visual check/tactile check, and confirmation of planned resection or preservation target (T)/confirmation of planned or ongoing resection line (L) accounted for 791/857, 885/763, 1148/500, 1208/440, and 1304/344 segments, respectively. The utterance's proportions of assistants, F, F of T on 3D-printed liver, F of T on real liver, and Pc of L on 3D-printed liver were significantly higher during non-expert surgeries than during expert surgeries. Confirming the surgical process with both 3D-printed liver and real liver and performing planning using a 3D-printed liver facilitates the safe implementation of TMRH, regardless of the surgeon's experience. CONCLUSIONS: The present study, using a unique conversation analysis, provided the first evidence for the clinical value of 3D-printed liver for TMRH for anatomical guidance of non-expert surgeons.

Anatomical attention can help to segment the dilated pancreatic duct in abdominal CT.

PURPOSE: Pancreatic duct dilation is associated with an increased risk of pancreatic cancer, the most lethal malignancy with the lowest 5-year relative survival rate. Automatic segmentation of the dilated pancreatic duct from contrast-enhanced CT scans would facilitate early diagnosis. However, pancreatic duct segmentation poses challenges due to its small anatomical structure and poor contrast in abdominal CT. In this work, we investigate an anatomical attention strategy to address this issue. METHODS: Our proposed anatomical attention strategy consists of two steps: pancreas localization and pancreatic duct segmentation. The coarse pancreatic mask segmentation is used to guide the fully convolutional networks (FCNs) to concentrate on the pancreas' anatomy and disregard unnecessary features. We further apply a multi-scale aggregation scheme to leverage the information from different scales. Moreover, we integrate the tubular structure enhancement as an additional input channel of FCN. RESULTS: We performed extensive experiments on 30 cases of contrast-enhanced abdominal CT volumes. To evaluate the pancreatic duct segmentation performance, we employed four measurements, including the Dice similarity coefficient (DSC), sensitivity, normalized surface distance, and 95 percentile Hausdorff distance. The average DSC achieves 55.7%, surpassing other pancreatic duct segmentation methods on single-phase CT scans only. CONCLUSIONS: We proposed an anatomical attention-based strategy for the dilated pancreatic duct segmentation. Our proposed strategy significantly outperforms earlier approaches. The attention mechanism helps to focus on the pancreas region, while the enhancement of the tubular structure enables FCNs to capture the vessel-like structure. The proposed technique might be applied to other tube-like structure segmentation tasks within targeted anatomies.

Artificial intelligence-based diagnostic imaging system with virtual enteroscopy and virtual unfolded views to evaluate small bowel lesions in Crohn's disease.

Since even subtle mucosal changes may be depicted using virtual endoscopy created by the three-dimensional reconstruction of MDCT images, we developed a novel diagnostic imaging system that integrates and displays virtual enteroscopy, curved planar reconstruction, and a virtual unfolded view, the width of which changes with increases/decreases in the inner luminal diameter. The system is also equipped with artificial intelligence that superimposes and displays depressed areas, generates an automatic small bowel centerline that connects fragmented small bowel regions, and performs electronic cleansing. We retrospectively evaluated the diagnostic performance of this system for small bowel lesions in Crohn's disease, which were divided into two groups: endoscopically-observable and endoscopically-unobservable. Lesion detection rates for stenoses, longitudinal ulcers with a cobblestone appearance, and scars were excellent in both groups. This system, when used in combination with endoscopy, shows slight mucosal changes in areas in which an endoscope cannot reach due to strictures, thereby extending the range of observation of the small bowel. This system is a useful diagnostic modality that has the capacity to assess mucosal healing and provide extraluminal information.

Deep learning model for extensive smartphone-based diagnosis and triage of cataracts and multiple corneal diseases.

AIM: To develop an artificial intelligence (AI) algorithm that diagnoses cataracts/corneal diseases from multiple conditions using smartphone images. METHODS: This study included 6442 images that were captured using a slit-lamp microscope (6106 images) and smartphone (336 images). An AI algorithm was developed based on slit-lamp images to differentiate 36 major diseases (cataracts and corneal diseases) into 9 categories. To validate the AI model, smartphone images were used for the testing dataset. We evaluated AI performance that included sensitivity, specificity and receiver operating characteristic (ROC) curve for the diagnosis and triage of the diseases. RESULTS: The AI algorithm achieved an area under the ROC curve of 0.998 (95% CI, 0.992 to 0.999) for normal eyes, 0.986 (95% CI, 0.978 to 0.997) for infectious keratitis, 0.960 (95% CI, 0.925 to 0.994) for immunological keratitis, 0.987 (95% CI, 0.978 to 0.996) for cornea scars, 0.997 (95% CI, 0.992 to 1.000) for ocular surface tumours, 0.993 (95% CI, 0.984 to 1.000) for corneal deposits, 1.000 (95% CI, 1.000 to 1.000) for acute angle-closure glaucoma, 0.992 (95% CI, 0.985 to 0.999) for cataracts and 0.993 (95% CI, 0.985 to 1.000) for bullous keratopathy. The triage of referral suggestion using the smartphone images exhibited high performance, in which the sensitivity and specificity were 1.00 (95% CI, 0.478 to 1.00) and 1.00 (95% CI, 0.976 to 1.000) for 'urgent', 0.867 (95% CI, 0.683 to 0.962) and 1.00 (95% CI, 0.971 to 1.000) for 'semi-urgent', 0.853 (95% CI, 0.689 to 0.950) and 0.983 (95% CI, 0.942 to 0.998) for 'routine' and 1.00 (95% CI, 0.958 to 1.00) and 0.896 (95% CI, 0.797 to 0.957) for 'observation', respectively. CONCLUSIONS: The AI system achieved promising performance in the diagnosis of cataracts and corneal diseases.

Artificial intelligence in a prediction model for postendoscopic retrograde cholangiopancreatography pancreatitis.

OBJECTIVES: In this study we aimed to develop an artificial intelligence-based model for predicting postendoscopic retrograde cholangiopancreatography (ERCP) pancreatitis (PEP). METHODS: We retrospectively reviewed ERCP patients at Nagoya University Hospital (NUH) and Toyota Memorial Hospital (TMH). We constructed two prediction models, a random forest (RF), one of the machine-learning algorithms, and a logistic regression (LR) model. First, we selected features of each model from 40 possible features. Then the models were trained and validated using three fold cross-validation in the NUH cohort and tested in the TMH cohort. The area under the receiver operating characteristic curve (AUROC) was used to assess model performance. Finally, using the output parameters of the RF model, we classified the patients into low-, medium-, and high-risk groups. RESULTS: A total of 615 patients at NUH and 544 patients at TMH were enrolled. Ten features were selected for the RF model, including albumin, creatinine, biliary tract cancer, pancreatic cancer, bile duct stone, total procedure time, pancreatic duct injection, pancreatic guidewire-assisted technique without a pancreatic stent, intraductal ultrasonography, and bile duct biopsy. In the three fold cross-validation, the RF model showed better predictive ability than the LR model (AUROC 0.821 vs. 0.660). In the test, the RF model also showed better performance (AUROC 0.770 vs. 0.663, P = 0.002). Based on the RF model, we classified the patients according to the incidence of PEP (2.9%, 10.0%, and 23.9%). CONCLUSION: We developed an RF model. Machine-learning algorithms could be powerful tools to develop accurate prediction models.

SGSR: style-subnets-assisted generative latent bank for large-factor super-resolution with registered medical image dataset.

PURPOSE: We propose a large-factor super-resolution (SR) method for performing SR on registered medical image datasets. Conventional SR approaches use low-resolution (LR) and high-resolution (HR) image pairs to train a deep convolutional neural network (DCN). However, LR-HR images in medical imaging are commonly acquired from different imaging devices, and acquiring LR-HR image pairs needs registration. Registered LR-HR images have registration errors inevitably. Using LR-HR images with registration error for training an SR DCN causes collapsed SR results. To address these challenges, we introduce a novel SR approach designed specifically for registered LR-HR medical images. METHODS: We propose style-subnets-assisted generative latent bank for large-factor super-resolution (SGSR) trained with registered medical image datasets. Pre-trained generative models named generative latent bank (GLB), which stores rich image priors, can be applied in SR to generate realistic and faithful images. We improve GLB by newly introducing style-subnets-assisted GLB (S-GLB). We also propose a novel inter-uncertainty loss to boost our method's performance. Introducing more spatial information by inputting adjacent slices further improved the results. RESULTS: SGSR outperforms state-of-the-art (SOTA) supervised SR methods qualitatively and quantitatively on multiple datasets. SGSR achieved higher reconstruction accuracy than recently supervised baselines by increasing peak signal-to-noise ratio from 32.628 to 34.206 dB. CONCLUSION: SGSR performs large-factor SR while given a registered LR-HR medical image dataset with registration error for training. SGSR's results have both realistic textures and accurate anatomical structures due to favorable quantitative and qualitative results. Experiments on multiple datasets demonstrated SGSR's superiority over other SOTA methods. SR medical images generated by SGSR are expected to improve the accuracy of pre-surgery diagnosis and reduce patient burden.

Automated Detection of the Thoracic Ossification of the Posterior Longitudinal Ligament Using Deep Learning and Plain Radiographs.

Ossification of the ligaments progresses slowly in the initial stages, and most patients are unaware of the disease until obvious myelopathy symptoms appear. Consequently, treatment and clinical outcomes are not satisfactory. This study is aimed at developing an automated system for the detection of the thoracic ossification of the posterior longitudinal ligament (OPLL) using deep learning and plain radiography. We retrospectively reviewed the data of 146 patients with thoracic OPLL and 150 control cases without thoracic OPLL. Plain lateral thoracic radiographs were used for object detection, training, and validation. Thereafter, an object detection system was developed, and its accuracy was calculated. The performance of the proposed system was compared with that of two spine surgeons. The accuracy of the proposed object detection model based on plain lateral thoracic radiographs was 83.4%, whereas the accuracies of spine surgeons 1 and 2 were 80.4% and 77.4%, respectively. Our findings indicate that our automated system, which uses a deep learning-based method based on plain radiographs, can accurately detect thoracic OPLL. This system has the potential to improve the diagnostic accuracy of thoracic OPLL.

Automated Detection and Diagnosis of Spinal Schwannomas and Meningiomas Using Deep Learning and Magnetic Resonance Imaging.

Spinal cord tumors are infrequently identified spinal diseases that are often difficult to diagnose even with magnetic resonance imaging (MRI) findings. To minimize the probability of overlooking these tumors and improve diagnostic accuracy, an automatic diagnostic system is needed. We aimed to develop an automated system for detecting and diagnosing spinal schwannomas and meningiomas based on deep learning using You Only Look Once (YOLO) version 4 and MRI. In this retrospective diagnostic accuracy study, the data of 50 patients with spinal schwannomas, 45 patients with meningiomas, and 100 control cases were reviewed, respectively. Sagittal T1-weighted (T1W) and T2-weighted (T2W) images were used for object detection, classification, training, and validation. The object detection and diagnosis system was developed using YOLO version 4. The accuracies of the proposed object detections based on T1W, T2W, and T1W + T2W images were 84.8%, 90.3%, and 93.8%, respectively. The accuracies of the object detection for two spine surgeons were 88.9% and 90.1%, respectively. The accuracies of the proposed diagnoses based on T1W, T2W, and T1W + T2W images were 76.4%, 83.3%, and 84.1%, respectively. The accuracies of the diagnosis for two spine surgeons were 77.4% and 76.1%, respectively. We demonstrated an accurate, automated detection and diagnosis of spinal schwannomas and meningiomas using the developed deep learning-based method based on MRI. This system could be valuable in supporting radiological diagnosis of spinal schwannomas and meningioma, with a potential of reducing the radiologist's overall workload.

Gaussian affinity and GIoU-based loss for perforation detection and localization from colonoscopy videos.

PURPOSE: Endoscopic submucosal dissection (ESD) is a minimally invasive treatment for early gastric cancer. However, perforations may happen and cause peritonitis during ESD. Thus, there is a potential demand for a computer-aided diagnosis system to support physicians in ESD. This paper presents a method to detect and localize perforations from colonoscopy videos to avoid perforation ignoring or enlarging by ESD physicians. METHOD: We proposed a training method for YOLOv3 by using GIoU and Gaussian affinity losses for perforation detection and localization in colonoscopic images. In this method, the object functional contains the generalized intersection over Union loss and Gaussian affinity loss. We propose a training method for the architecture of YOLOv3 with the presented loss functional to detect and localize perforations precisely. RESULTS: To qualitatively and quantitatively evaluate the presented method, we created a dataset from 49 ESD videos. The results of the presented method on our dataset revealed a state-of-the-art performance of perforation detection and localization, which achieved 0.881 accuracy, 0.869 AUC, and 0.879 mean average precision. Furthermore, the presented method is able to detect a newly appeared perforation in 0.1 s. CONCLUSIONS: The experimental results demonstrated that YOLOv3 trained by the presented loss functional were very effective in perforation detection and localization. The presented method can quickly and precisely remind physicians of perforation happening in ESD. We believe a future CAD system can be constructed for clinical applications with the proposed method.

Deep learning-based prediction model for postoperative complications of cervical posterior longitudinal ligament ossification.

PURPOSE: Postoperative complication prediction helps surgeons to inform and manage patient expectations. Deep learning, a model that finds patterns in large samples of data, outperform traditional statistical methods in making predictions. This study aimed to create a deep learning-based model (DLM) to predict postoperative complications in patients with cervical ossification of the posterior longitudinal ligament (OPLL). METHODS: This prospective multicenter study was conducted by the 28 institutions, and 478 patients were included in the analysis. Deep learning was used to create two predictive models of the overall postoperative complications and neurological complications, one of the major complications. These models were constructed by learning the patient's preoperative background, clinical symptoms, surgical procedures, and imaging findings. These logistic regression models were also created, and these accuracies were compared with those of the DLM. RESULTS: Overall complications were observed in 127 cases (26.6%). The accuracy of the DLM was 74.6 ± 3.7% for predicting the overall occurrence of complications, which was comparable to that of the logistic regression (74.1%). Neurological complications were observed in 48 cases (10.0%), and the accuracy of the DLM was 91.7 ± 3.5%, which was higher than that of the logistic regression (90.1%). CONCLUSION: A new algorithm using deep learning was able to predict complications after cervical OPLL surgery. This model was well calibrated, with prediction accuracy comparable to that of regression models. The accuracy remained high even for predicting only neurological complications, for which the case number is limited compared to conventional statistical methods.

Class-wise confidence-aware active learning for laparoscopic images segmentation.

PURPOSE: Segmentation tasks are important for computer-assisted surgery systems as they provide the shapes of organs and the locations of instruments. What prevents the most powerful segmentation approaches from becoming practical applications is the requirement for annotated data. Active learning provides strategies to dynamically select the most informative samples to reduce the annotation workload. However, most previous active learning literature has failed to select the frames that containing low-appearing frequency classes, even though the existence of these classes is common in laparoscopic videos, resulting in poor performance in segmentation tasks. Furthermore, few previous works have explored the unselected data to improve active learning. Therefore, in this work, we focus on these classes to improve the segmentation performance. METHODS: We propose a class-wise confidence bank that stores and updates the confidence scores for each class and a new acquisition function based on a confidence bank. We apply confidence scores to explore an unlabeled dataset by combining it with a class-wise data mixture method to exploit unlabeled datasets without any annotation. RESULTS: We validated our proposal on two open-source datasets, CholecSeg8k and RobSeg2017, and observed that its performance surpassed previous active learning studies with about [Formula: see text] improvement on CholecSeg8k, especially for classes with a low-appearing frequency. For robSeg2017, we conducted experiments with a small and large annotation budgets to validate situation that shows the effectiveness of our proposal. CONCLUSIONS: We presented a class-wise confidence score to improve the acquisition function for active learning and explored unlabeled data with our proposed class-wise confidence score, which results in a large improvement over the compared methods. The experiments also showed that our proposal improved the segmentation performance for classes with a low-appearing frequency.

A skeleton context-aware 3D fully convolutional network for abdominal artery segmentation.

PURPOSE: This paper aims to propose a deep learning-based method for abdominal artery segmentation. Blood vessel structure information is essential to diagnosis and treatment. Accurate blood vessel segmentation is critical to preoperative planning. Although deep learning-based methods perform well on large organs, segmenting small organs such as blood vessels is challenging due to complicated branching structures and positions. We propose a 3D deep learning network from a skeleton context-aware perspective to improve segmentation accuracy. In addition, we propose a novel 3D patch generation method which could strengthen the structural diversity of a training data set. METHOD: The proposed method segments abdominal arteries from an abdominal computed tomography (CT) volume using a 3D fully convolutional network (FCN). We add two auxiliary tasks to the network to extract the skeleton context of abdominal arteries. In addition, our skeleton-based patch generation (SBPG) method further enables the FCN to segment small arteries. SBPG generates a 3D patch from a CT volume by leveraging artery skeleton information. These methods improve the segmentation accuracies of small arteries. RESULTS: We used 20 cases of abdominal CT volumes to evaluate the proposed method. The experimental results showed that our method outperformed previous segmentation accuracies. The averaged precision rate, recall rate, and F-measure were 95.5%, 91.0%, and 93.2%, respectively. Compared to a baseline method, our method improved 1.5% the averaged recall rate and 0.7% the averaged F-measure. CONCLUSIONS: We present a skeleton context-aware 3D FCN to segment abdominal arteries from an abdominal CT volume. In addition, we propose a 3D patch generation method. Our fully automated method segmented most of the abdominal artery regions. The method produced competitive segmentation performance compared to previous methods.

Classification and visual explanation for COVID-19 pneumonia from CT images using triple learning.

This study presents a novel framework for classifying and visualizing pneumonia induced by COVID-19 from CT images. Although many image classification methods using deep learning have been proposed, in the case of medical image fields, standard classification methods are unable to be used in some cases because the medical images that belong to the same category vary depending on the progression of the symptoms and the size of the inflamed area. In addition, it is essential that the models used be transparent and explainable, allowing health care providers to trust the models and avoid mistakes. In this study, we propose a classification method using contrastive learning and an attention mechanism. Contrastive learning is able to close the distance for images of the same category and generate a better feature space for classification. An attention mechanism is able to emphasize an important area in the image and visualize the location related to classification. Through experiments conducted on two-types of classification using a three-fold cross validation, we confirmed that the classification accuracy was significantly improved; in addition, a detailed visual explanation was achieved comparison with conventional methods.

Positive-gradient-weighted object activation mapping: visual explanation of object detector towards precise colorectal-polyp localisation.

PURPOSE: Precise polyp detection and localisation are essential for colonoscopy diagnosis. Statistical machine learning with a large-scale data set can contribute to the construction of a computer-aided diagnosis system for the prevention of overlooking and miss-localisation of a polyp in colonoscopy. We propose new visual explaining methods for a well-trained object detector, which achieves fast and accurate polyp detection with a bounding box towards a precise automated polyp localisation. METHOD: We refine gradient-weighted class activation mapping for more accurate highlighting of important patterns in processing a convolutional neural network. Extending the refined mapping into multiscaled processing, we define object activation mapping that highlights important object patterns in an image for a detection task. Finally, we define polyp activation mapping to achieve precise polyp localisation by integrating adaptive local thresholding into object activation mapping. We experimentally evaluate the proposed visual explaining methods with four publicly available databases. RESULTS: The refined mapping visualises important patterns in each convolutional layer more accurately than the original gradient-weighted class activation mapping. The object activation mapping clearly visualises important patterns in colonoscopic images for polyp detection. The polyp activation mapping localises the detected polyps in ETIS-Larib, CVC-Clinic and Kvasir-SEG database with mean Dice scores of 0.76, 0.72 and 0.72, respectively. CONCLUSIONS: We developed new visual explaining methods for a convolutional neural network by refining and extending gradient-weighted class activation mapping. Experimental results demonstrated the validity of the proposed methods by showing that accurate visualisation of important patterns and localisation of polyps in a colonoscopic image. The proposed visual explaining methods are useful for the interpreting and applying a trained polyp detector.

SR-CycleGAN: super-resolution of clinical CT to micro-CT level with multi-modality super-resolution loss.

Purpose: We propose a super-resolution (SR) method, named SR-CycleGAN, for SR of clinical computed tomography (CT) images to the micro-focus x-ray CT CT ( µ CT ) level. Due to the resolution limitations of clinical CT (about 500 × 500 × 500 µ m 3 / voxel ), it is challenging to obtain enough pathological information. On the other hand, µ CT scanning allows the imaging of lung specimens with significantly higher resolution (about 50 × 50 × 50 µ m 3 / voxel or higher), which allows us to obtain and analyze detailed anatomical information. As a way to obtain detailed information such as cancer invasion and bronchioles from preoperative clinical CT images of lung cancer patients, the SR of clinical CT images to the µ CT level is desired. Approach: Typical SR methods require aligned pairs of low-resolution (LR) and high-resolution images for training, but it is infeasible to obtain precisely aligned paired clinical CT and µ CT images. To solve this problem, we propose an unpaired SR approach that can perform SR on clinical CT to the µ CT level. We modify a conventional image-to-image translation network named CycleGAN to an inter-modality translation network named SR-CycleGAN. The modifications consist of three parts: (1) an innovative loss function named multi-modality super-resolution loss, (2) optimized SR network structures for enlarging the input LR image to 2 k -times by width and height to obtain the SR output, and (3) sub-pixel shuffling layers for reducing computing time. Results: Experimental results demonstrated that our method successfully performed SR of lung clinical CT images. SSIM and PSNR scores of our method were 0.54 and 17.71, higher than the conventional CycleGAN's scores of 0.05 and 13.64, respectively. Conclusions: The proposed SR-CycleGAN is usable for the SR of a lung clinical CT into µ CT scale, while conventional CycleGAN output images with low qualitative and quantitative values. More lung micro-anatomy information could be observed to aid diagnosis, such as the shape of bronchioles walls.