Weakly supervised histopathology image segmentation with self-attention.

Accurate segmentation in histopathology images at pixel-level plays a critical role in the digital pathology workflow. The development of weakly supervised methods for histopathology image segmentation liberates pathologists from time-consuming and labor-intensive works, opening up possibilities of further automated quantitative analysis of whole-slide histopathology images. As an effective subgroup of weakly supervised methods, multiple instance learning (MIL) has achieved great success in histopathology images. In this paper, we specially treat pixels as instances so that the histopathology image segmentation task is transformed into an instance prediction task in MIL. However, the lack of relations between instances in MIL limits the further improvement of segmentation performance. Therefore, we propose a novel weakly supervised method called SA-MIL for pixel-level segmentation in histopathology images. SA-MIL introduces a self-attention mechanism into the MIL framework, which captures global correlation among all instances. In addition, we use deep supervision to make the best use of information from limited annotations in the weakly supervised method. Our approach makes up for the shortcoming that instances are independent of each other in MIL by aggregating global contextual information. We demonstrate state-of-the-art results compared to other weakly supervised methods on two histopathology image datasets. It is evident that our approach has generalization ability for the high performance on both tissue and cell histopathology datasets. There is potential in our approach for various applications in medical images.

Evaluation of free DIEP flap complications in the setting of previous gynecological operations.

BACKGROUND: Autologous tissue breast reconstruction with free deep inferior epigastric perforator (DIEP) flaps is reliable with reproducible results and very few contraindications. However, previous surgery may compromise the abdominal donor site due to injury to the vascular pedicle. The purpose of the current study is to evaluate the effects of prior abdominal surgery on need for changes to the operative plan, intraoperative complications, and postoperative flap compromise. PATIENTS AND METHODS: A retrospective review of all patients undergoing breast reconstruction with free tissue transfer from the abdomen was performed. RESULTS: A total of 733 free abdominal flaps were performed in 478 patients during the study period. Two hundred sixty-two (54.8%) patients had prior abdominal surgery with 24.8% laparoscopic/robotic versus 56.9% open versus 18.3% both, 21.4% general surgery versus 60.7% gynecological versus 17.9% both, and 97.7% elective versus 1.1% emergent versus 1.1% both. There were 15 total flap losses (2.0%) and 2 partial flap losses (0.3%). Intraoperative complications and changes in the operative plan occurred in 13 flaps (1.8%) with 84.6% having prior gynecological surgery (p = .0001). CONCLUSIONS: Free DIEP flap breast reconstruction is becoming more commonplace with a low risk of complications. Although DIEP flaps are still possible in the setting of prior abdominal surgery, there is a higher risk of damage to the deep inferior epigastric pedicle in patients who have had emergency Cesarean sections or hysterectomy. Conducting a focused history may prepare the reconstructive microsurgeon to address and to avoid potential intraoperative complications.

Narrative review of patient-reported outcomes of breast reconstruction.

Background and Objective: The treatment of breast cancer encompasses both the elimination of malignancy as well as reconstruction after tumor extirpation. Although the patient may have had successful treatment of her breast cancer, the resulting disfigurement and deformity can have a substantial impact on her physical and mental well-being. Breast reconstruction affords these patients the opportunity to correct these deformities and potentially to improve their quality of life. The current literature review evaluates patient-reported outcomes for the various options of breast reconstruction that are most commonly performed. Methods: A literature review on PubMed with the key words "patient-reported outcomes", "breast reconstruction", and "breast cancer" yielded 738 results, which were screened. Articles that specifically focused on patient-reported outcomes after various types of breast reconstruction were evaluated and included in this literature review. Key Content and Findings: The main options of alloplastic reconstruction, autologous tissue reconstruction, and oncoplastic reconstruction were reviewed and found to demonstrate high levels of patient satisfaction. Although there is no clear superior option, patient-reported outcomes demonstrate improved well-being compared to no reconstruction. Conclusions: Breast reconstruction provides the opportunity to correct the deformities after breast cancer treatment making it a crucial component of comprehensive cancer care. A variety of reconstructive options are available which can be tailored to each individual patient to achieve the optimal results for that particular patient. Therefore patient-reported outcomes are paramount to gauge the true success of not only breast cancer treatment but also reconstructive aspects after treatment.

Integrated 3d flow-based multi-atlas brain structure segmentation.

MRI brain structure segmentation plays an important role in neuroimaging studies. Existing methods either spend much CPU time, require considerable annotated data, or fail in segmenting volumes with large deformation. In this paper, we develop a novel multi-atlas-based algorithm for 3D MRI brain structure segmentation. It consists of three modules: registration, atlas selection and label fusion. Both registration and label fusion leverage an integrated flow based on grayscale and SIFT features. We introduce an effective and efficient strategy for atlas selection by employing the accompanying energy generated in the registration step. A 3D sequential belief propagation method and a 3D coarse-to-fine flow matching approach are developed in both registration and label fusion modules. The proposed method is evaluated on five public datasets. The results show that it has the best performance in almost all the settings compared to competitive methods such as ANTs, Elastix, Learning to Rank and Joint Label Fusion. Moreover, our registration method is more than 7 times as efficient as that of ANTs SyN, while our label transfer method is 18 times faster than Joint Label Fusion in CPU time. The results on the ADNI dataset demonstrate that our method is applicable to image pairs that require a significant transformation in registration. The performance on a composite dataset suggests that our method succeeds in a cross-modality manner. The results of this study show that the integrated 3D flow-based method is effective and efficient for brain structure segmentation. It also demonstrates the power of SIFT features, multi-atlas segmentation and classical machine learning algorithms for a medical image analysis task. The experimental results on public datasets show the proposed method's potential for general applicability in various brain structures and settings.

3D Segmentation Guided Style-Based Generative Adversarial Networks for PET Synthesis.

Potential radioactive hazards in full-dose positron emission tomography (PET) imaging remain a concern, whereas the quality of low-dose images is never desirable for clinical use. So it is of great interest to translate low-dose PET images into full-dose. Previous studies based on deep learning methods usually directly extract hierarchical features for reconstruction. We notice that the importance of each feature is different and they should be weighted dissimilarly so that tiny information can be captured by the neural network. Furthermore, the synthesis on some regions of interest is important in some applications. Here we propose a novel segmentation guided style-based generative adversarial network (SGSGAN) for PET synthesis. (1) We put forward a style-based generator employing style modulation, which specifically controls the hierarchical features in the translation process, to generate images with more realistic textures. (2) We adopt a task-driven strategy that couples a segmentation task with a generative adversarial network (GAN) framework to improve the translation performance. Extensive experiments show the superiority of our overall framework in PET synthesis, especially on those regions of interest.

Whole brain segmentation with full volume neural network.

Whole brain segmentation is an important neuroimaging task that segments the whole brain volume into anatomically labeled regions-of-interest. Convolutional neural networks have demonstrated good performance in this task. Existing solutions, usually segment the brain image by classifying the voxels, or labeling the slices or the sub-volumes separately. Their representation learning is based on parts of the whole volume whereas their labeling result is produced by aggregation of partial segmentation. Learning and inference with incomplete information could lead to sub-optimal final segmentation result. To address these issues, we propose to adopt a full volume framework, which feeds the full volume brain image into the segmentation network and directly outputs the segmentation result for the whole brain volume. The framework makes use of complete information in each volume and can be implemented easily. An effective instance in this framework is given subsequently. We adopt the 3D high-resolution network (HRNet) for learning spatially fine-grained representations and the mixed precision training scheme for memory-efficient training. Extensive experiment results on a publicly available 3D MRI brain dataset show that our proposed model advances the state-of-the-art methods in terms of segmentation performance.

Phrenic Nerve Reconstruction for Effective Surgical Treatment of Diaphragmatic Paralysis.

ABSTRACT: Diaphragmatic paralysis due to phrenic nerve injury may cause orthopnea, exertional dyspnea, and sleep-disordered breathing. Phrenic nerve reconstruction may relieve symptoms and improve respiratory function. A retrospective review of 400 consecutive patients undergoing phrenic nerve reconstruction for diaphragmatic paralysis at 2 tertiary treatment centers was performed between 2007 and 2019. Symptomatic patients were identified, and the diagnosis was confirmed on radiographic evaluations. Assessment parameters included pulmonary spirometry (forced expiratory volume in 1 second and FVC), maximal inspiratory pressure, compound muscle action potentials, diaphragm thickness, chest fluoroscopy, and Short Form 36 Health Survey Questionnaire (SF-36) survey. There were 81 females and 319 males with an average age of 54 years (range, 19-79 years). The mean duration from diagnosis to surgery was 29 months (range, 1-320 months). The most common etiologies were acute or chronic injury (29%), interscalene nerve block (17%), and cardiothoracic surgery (15%). The mean improvements in forced expiratory volume in 1 second and FVC at 1 year were 10% (P < 0.01) and 8% (P < 0.05), respectively. At 2-year follow-up, the corresponding values were 22% (P < 0.05) and 18% (P < 0.05), respectively. Improvement on chest fluoroscopy was demonstrated in 63% and 71% of patients at 1 and 2-year follow-up, respectively. There was a 20% (P < 0.01) improvement in maximal inspiratory pressure, and compound muscle action potentials increased by 82% (P < 0.001). Diaphragm thickness demonstrated a 27% (P < 0.01) increase, and SF-36 revealed a 59% (P < 0.001) improvement in physical functioning. Symptomatic diaphragmatic paralysis should be considered for surgical treatment. Phrenic nerve reconstruction can achieve symptomatic relief and improve respiratory function. Increasing spirometry and improvements on Sniff from 1 to 2 years support incremental recovery with longer follow-up.

Treatment for bilateral diaphragmatic dysfunction using phrenic nerve reconstruction and diaphragm pacemakers.

OBJECTIVES: Bilateral diaphragmatic dysfunction results in severe dyspnoea, usually requiring oxygen therapy and nocturnal ventilatory support. Although treatment options are limited, phrenic nerve reconstruction (PR) offers the opportunity to restore functional activity. This study aims to evaluate combination treatment with PR and placement of a diaphragm pacemaker (DP) compared to DP placement alone in patients with bilateral diaphragmatic dysfunction. METHODS: Patients with bilateral diaphragmatic dysfunction were prospectively enrolled in the following treatment algorithm: Unilateral PR was performed on the more severely impacted side with bilateral DP implantation. Motor amplitudes, ultrasound measurements of diaphragm thickness, maximal inspiratory pressure, forced expiratory volume, forced vital capacity and subjective patient-reported outcomes were obtained for retrospective analysis following completion of the prospective database. RESULTS: Fourteen male patients with bilateral diaphragmatic dysfunction confirmed on chest fluoroscopy and electrodiagnostic testing were included. All 14 patients required nocturnal ventilator support, and 8/14 (57.1%) were oxygen-dependent. All patients reported subjective improvement, and all 8 oxygen-dependent patients were able to discontinue oxygen therapy following treatment. Improvements in maximal inspiratory pressure, forced vital capacity and forced expiratory volume were 68%, 47% and 53%, respectively. There was an average improvement of 180% in motor amplitude and a 50% increase in muscle thickness. Comparison of motor amplitude changes revealed significantly greater functional recovery on the PR + DP side. CONCLUSIONS: PR and simultaneous implantation of a DP may restore functional activity and alleviate symptoms in patients with bilateral diaphragmatic dysfunction. PR plus diaphragm pacing appear to result in greater functional muscle recovery than pacing alone.

Deep learning in digital pathology image analysis: a survey.

Deep learning (DL) has achieved state-of-the-art performance in many digital pathology analysis tasks. Traditional methods usually require hand-crafted domain-specific features, and DL methods can learn representations without manually designed features. In terms of feature extraction, DL approaches are less labor intensive compared with conventional machine learning methods. In this paper, we comprehensively summarize recent DL-based image analysis studies in histopathology, including different tasks (e.g., classification, semantic segmentation, detection, and instance segmentation) and various applications (e.g., stain normalization, cell/gland/region structure analysis). DL methods can provide consistent and accurate outcomes. DL is a promising tool to assist pathologists in clinical diagnosis.

ANHIR: Automatic Non-Rigid Histological Image Registration Challenge.

Automatic Non-rigid Histological Image Registration (ANHIR) challenge was organized to compare the performance of image registration algorithms on several kinds of microscopy histology images in a fair and independent manner. We have assembled 8 datasets, containing 355 images with 18 different stains, resulting in 481 image pairs to be registered. Registration accuracy was evaluated using manually placed landmarks. In total, 256 teams registered for the challenge, 10 submitted the results, and 6 participated in the workshop. Here, we present the results of 7 well-performing methods from the challenge together with 6 well-known existing methods. The best methods used coarse but robust initial alignment, followed by non-rigid registration, used multiresolution, and were carefully tuned for the data at hand. They outperformed off-the-shelf methods, mostly by being more robust. The best methods could successfully register over 98% of all landmarks and their mean landmark registration accuracy (TRE) was 0.44% of the image diagonal. The challenge remains open to submissions and all images are available for download.

MRI Cross-Modality Image-to-Image Translation.

We present a cross-modality generation framework that learns to generate translated modalities from given modalities in MR images. Our proposed method performs Image Modality Translation (abbreviated as IMT) by means of a deep learning model that leverages conditional generative adversarial networks (cGANs). Our framework jointly exploits the low-level features (pixel-wise information) and high-level representations (e.g. brain tumors, brain structure like gray matter, etc.) between cross modalities which are important for resolving the challenging complexity in brain structures. Our framework can serve as an auxiliary method in medical use and has great application potential. Based on our proposed framework, we first propose a method for cross-modality registration by fusing the deformation fields to adopt the cross-modality information from translated modalities. Second, we propose an approach for MRI segmentation, translated multichannel segmentation (TMS), where given modalities, along with translated modalities, are segmented by fully convolutional networks (FCN) in a multichannel manner. Both of these two methods successfully adopt the cross-modality information to improve the performance without adding any extra data. Experiments demonstrate that our proposed framework advances the state-of-the-art on five brain MRI datasets. We also observe encouraging results in cross-modality registration and segmentation on some widely adopted brain datasets. Overall, our work can serve as an auxiliary method in medical use and be applied to various tasks in medical fields.

Unsupervised 3D End-to-End Medical Image Registration With Volume Tweening Network.

3D medical image registration is of great clinical importance. However, supervised learning methods require a large amount of accurately annotated corresponding control points (or morphing), which are very difficult to obtain. Unsupervised learning methods ease the burden of manual annotation by exploiting unlabeled data without supervision. In this article, we propose a new unsupervised learning method using convolutional neural networks under an end-to-end framework, Volume Tweening Network (VTN), for 3D medical image registration. We propose three innovative technical components: (1) An end-to-end cascading scheme that resolves large displacement; (2) An efficient integration of affine registration network; and (3) An additional invertibility loss that encourages backward consistency. Experiments demonstrate that our algorithm is 880x faster (or 3.3x faster without GPU acceleration) than traditional optimization-based methods and achieves state-of-the-art performance in medical image registration.

Comparison of Phasix, polypropylene, and primary closure of the abdominal donor site after bilateral free flap breast reconstruction: Long-term evaluation of abdominal hernia and bulge formation.

BACKGROUND: Abdominal free flap harvest for breast reconstruction may result in significant morbidity in terms of hernias and bulges. Reinforcement of the donor site with mesh has been recommended to minimize the risk of hernias and bulges, but no studies exist evaluating the optimal type of mesh. Polypropylene has traditionally been used but the development of Phasix restorable mesh may be a reasonable alternative. Here, we compared the use of Phasix to polypropylene and primary closure and hypothesize that the former has lower rates of abdominal morbidity in the long term. PATIENTS AND METHODS: A retrospective review of all patients undergoing bilateral free flap breast reconstruction from the abdomen was performed while patients with pedicle flaps or alternative donor sites were excluded. Patient demographics, medical/surgical history, cancer treatments, and flap type were analyzed. All patients were monitored for a minimum of 2 years for early donor site complications as well as hernia/bulges. RESULTS: Sixty-six consecutive patients were included (40 patients with Phasix, 20 patients with polypropylene, and 6 patients with primary closure). Use of Phasix mesh resulted in higher initial operative costs ($2,750 vs. $72 vs. $0). Two patients with polypropylene mesh and one patient undergoing primary closure developed an abdominal bulge in an average follow-up of 25.2 months (11.5% vs. 0%, p = .04). CONCLUSIONS: Mesh placement for abdominal wall reinforcement after bilateral free flap breast reconstruction minimizes the risk of hernias and bulges. Although Phasix results in increased initial costs, abdominal morbidity is significantly decreased after follow-up beyond 2 years.

Wound area measurement with 3D transformation and smartphone images.

BACKGROUND: Quantitative areas is of great measurement of wound significance in clinical trials, wound pathological analysis, and daily patient care. 2D methods cannot solve the problems caused by human body curvatures and different camera shooting angles. Our objective is to simply collect wound areas, accurately measure wound areas and overcome the shortcomings of 2D methods. RESULTS: We propose a method with 3D transformation to measure wound area on a human body surface, which combines structure from motion (SFM), least squares conformal mapping (LSCM), and image segmentation. The method captures 2D images of wound, which is surrounded by adhesive tape scale next to it, by smartphone and implements 3D reconstruction from the images based on SFM. Then it uses LSCM to unwrap the UV map of the 3D model. In the end, it utilizes image segmentation by interactive method for wound extraction and measurement. Our system yields state-of-the-art results on a dataset of 118 wounds on 54 patients, and performs with an accuracy of 0.97. The Pearson correlation, standardized regression coefficient and adjusted R square of our method are 0.999, 0.895 and 0.998 respectively. CONCLUSIONS: A smartphone is used to capture wound images, which lowers costs, lessens dependence on hardware, and avoids the risk of infection. The quantitative calculation of the 3D wound area is realized, solving the challenges that 2D methods cannot and achieving a good accuracy.

Mapping anatomical related entities to human body parts based on wikipedia in discharge summaries.

*: Background Consisting of dictated free-text documents such as discharge summaries, medical narratives are widely used in medical natural language processing. Relationships between anatomical entities and human body parts are crucial for building medical text mining applications. To achieve this, we establish a mapping system consisting of a Wikipedia-based scoring algorithm and a named entity normalization method (NEN). The mapping system makes full use of information available on Wikipedia, which is a comprehensive Internet medical knowledge base. We also built a new ontology, Tree of Human Body Parts (THBP), from core anatomical parts by referring to anatomical experts and Unified Medical Language Systems (UMLS) to make the mapping system efficacious for clinical treatments. *: Result The gold standard is derived from 50 discharge summaries from our previous work, in which 2,224 anatomical entities are included. The F1-measure of the baseline system is 70.20%, while our algorithm based on Wikipedia achieves 86.67% with the assistance of NEN. *: Conclusions We construct a framework to map anatomical entities to THBP ontology using normalization and a scoring algorithm based on Wikipedia. The proposed framework is proven to be much more effective and efficient than the main baseline system.

Predicting breast tumor proliferation from whole-slide images: The TUPAC16 challenge.

Tumor proliferation is an important biomarker indicative of the prognosis of breast cancer patients. Assessment of tumor proliferation in a clinical setting is a highly subjective and labor-intensive task. Previous efforts to automate tumor proliferation assessment by image analysis only focused on mitosis detection in predefined tumor regions. However, in a real-world scenario, automatic mitosis detection should be performed in whole-slide images (WSIs) and an automatic method should be able to produce a tumor proliferation score given a WSI as input. To address this, we organized the TUmor Proliferation Assessment Challenge 2016 (TUPAC16) on prediction of tumor proliferation scores from WSIs. The challenge dataset consisted of 500 training and 321 testing breast cancer histopathology WSIs. In order to ensure fair and independent evaluation, only the ground truth for the training dataset was provided to the challenge participants. The first task of the challenge was to predict mitotic scores, i.e., to reproduce the manual method of assessing tumor proliferation by a pathologist. The second task was to predict the gene expression based PAM50 proliferation scores from the WSI. The best performing automatic method for the first task achieved a quadratic-weighted Cohen's kappa score of κ = 0.567, 95% CI [0.464, 0.671] between the predicted scores and the ground truth. For the second task, the predictions of the top method had a Spearman's correlation coefficient of r = 0.617, 95% CI [0.581 0.651] with the ground truth. This was the first comparison study that investigated tumor proliferation assessment from WSIs. The achieved results are promising given the difficulty of the tasks and weakly-labeled nature of the ground truth. However, further research is needed to improve the practical utility of image analysis methods for this task.

Interrater agreement between American and Chinese sleep centers according to the 2014 AASM standard.

OBJECTIVES: To determine inter-lab reliability in sleep stage scoring using the 2014 American Academy of Sleep Medicine (AASM) manual. To understand in-depth reasons for disagreement and provide suggestions for improvement. METHODS: This study consisted of 40 all-night polysomnographys (PSGs) from different samples. PSGs were segmented into 37,642 30-s epochs. Five doctors from China and two doctors from America scored the epochs following the 2014 AASM standard. Scoring disagreement between two centers was evaluated using Cohen's kappa (κ). After visual inspection of PSGs of deviating scorings, potential disagreement reasons were analyzed. RESULTS: Inter-lab reliability yielded a substantial degree (κ = 0.75 ± 0.01). Scoring for stage W (κ = 0.89) and R (κ = 0.87) achieved the highest agreement, while stage N1 (κ = 0.45) reflected the lowest. Considering the relative disagreement ratio, N2-N3 (22.09%), W-N1 (19.68%), and N1-N2 (18.75%) were the most frequent combinations of discrepancy. American and Chinese doctors showed certain characteristics in the scoring of discrepancy combination W-N1, N1-N2, and N2-N3. There are seven reasons for disagreement, namely "on-threshold characteristic" (29.21%), "context influence" (18.06%), "characteristic identification difficulty" (8.81%), "arousal-wake confusion" (7.57%), "derivation inconsistence" (2.15%), "on-borderline characteristic" (0.92%), and "misrecognition" (33.27%). CONCLUSIONS: This study demonstrated the sleep stage scoring agreement of the 2014 AASM manual and explored potential sources of labeling ambiguity. Improvement measures were suggested accordingly to help remove ambiguity for scorers and improve scoring reliability at the international level.

Unsupervised Learning for Cell-Level Visual Representation in Histopathology Images With Generative Adversarial Networks.

The visual attributes of cells, such as the nuclear morphology and chromatin openness, are critical for histopathology image analysis. By learning cell-level visual representation, we can obtain a rich mix of features that are highly reusable for various tasks, such as cell-level classification, nuclei segmentation, and cell counting. In this paper, we propose a unified generative adversarial networks architecture with a new formulation of loss to perform robust cell-level visual representation learning in an unsupervised setting. Our model is not only label-free and easily trained but also capable of cell-level unsupervised classification with interpretable visualization, which achieves promising results in the unsupervised classification of bone marrow cellular components. Based on the proposed cell-level visual representation learning, we further develop a pipeline that exploits the varieties of cellular elements to perform histopathology image classification, the advantages of which are demonstrated on bone marrow datasets.