Precise immunofluorescence canceling for highly multiplexed imaging to capture specific cell states.

Cell states are regulated by the response of signaling pathways to receptor ligand-binding and intercellular interactions. High-resolution imaging has been attempted to explore the dynamics of these processes and, recently, multiplexed imaging has profiled cell states by achieving a comprehensive acquisition of spatial protein information from cells. However, the specificity of antibodies is still compromised when visualizing activated signals. Here, we develop Precise Emission Canceling Antibodies (PECAbs) that have cleavable fluorescent labeling. PECAbs enable high-specificity sequential imaging using hundreds of antibodies, allowing for reconstruction of the spatiotemporal dynamics of signaling pathways. Additionally, combining this approach with seq-smFISH can effectively classify cells and identify their signal activation states in human tissue. Overall, the PECAb system can serve as a comprehensive platform for analyzing complex cell processes.

Development of an automatic surgical planning system for high tibial osteotomy using artificial intelligence.

BACKGROUND: This study proposed an automatic surgical planning system for high tibial osteotomy (HTO) using deep learning-based artificial intelligence and validated its accuracy. The system simulates osteotomy and measures lower-limb alignment parameters in pre- and post-osteotomy simulations. METHODS: A total of 107 whole-leg standing radiographs were obtained from 107 patients who underwent HTO. First, the system detected anatomical landmarks on radiographs. Then, it simulated osteotomy and automatically measured five parameters in pre- and post-osteotomy simulation (hip knee angle [HKA], weight-bearing line ratio [WBL ratio], mechanical lateral distal femoral angle [mLDFA], mechanical medial proximal tibial angle [mMPTA], and mechanical lateral distal tibial angle [mLDTA]). The accuracy of the measured parameters was validated by comparing them with the ground truth (GT) values given by two orthopaedic surgeons. RESULTS: All absolute errors of the system were within 1.5° or 1.5%. All inter-rater correlation confidence (ICC) values between the system and GT showed good reliability (>0.80). Excellent reliability was observed in the HKA (0.99) and WBL ratios (>0.99) for the pre-osteotomy simulation. The intra-rater difference of the system exhibited excellent reliability with an ICC value of 1.00 for all lower-limb alignment parameters in pre- and post-osteotomy simulations. In addition, the measurement time per radiograph (0.24 s) was considerably shorter than that of an orthopaedic surgeon (118 s). CONCLUSION: The proposed system is practically applicable because it can measure lower-limb alignment parameters accurately and quickly in pre- and post-osteotomy simulations. The system has potential applications in surgical planning systems.

A Deep Learning-Based Assay for Programmed Death Ligand 1 Immunohistochemistry Scoring in Non-Small Cell Lung Carcinoma: Does it Help Pathologists Score?

Several studies have developed various artificial intelligence (AI) models for immunohistochemical analysis of programmed death ligand 1 (PD-L1) in patients with non-small cell lung carcinoma; however, none have focused on specific ways by which AI-assisted systems could help pathologists determine the tumor proportion score (TPS). In this study, we developed an AI model to calculate the TPS of the PD-L1 22C3 assay and evaluated whether and how this AI-assisted system could help pathologists determine the TPS and analyze how AI-assisted systems could affect pathologists' assessment accuracy. We assessed the 4 methods of the AI-assisted system: (1 and 2) pathologists first assessed and then referred to automated AI scoring results (1, positive tumor cell percentage; 2, positive tumor cell percentage and visualized overlay image) for final confirmation, and (3 and 4) pathologists referred to the automated AI scoring results (3, positive tumor cell percentage; 4, positive tumor cell percentage and visualized overlay image) while determining TPS. Mixed-model analysis was used to calculate the odds ratios (ORs) with 95% CI for AI-assisted TPS methods 1 to 4 compared with pathologists' scoring. For all 584 samples of the tissue microarray, the OR for AI-assisted TPS methods 1 to 4 was 0.94 to 1.07 and not statistically significant. Of them, we found 332 discordant cases, on which the pathologists' judgments were inconsistent; the ORs for AI-assisted TPS methods 1, 2, 3, and 4 were 1.28 (1.06-1.54; P = .012), 1.29 (1.06-1.55; P = .010), 1.28 (1.06-1.54; P = .012), and 1.29 (1.06-1.55; P = .010), respectively, which were statistically significant. For discordant cases, the OR for each AI-assisted TPS method compared with the others was 0.99 to 1.01 and not statistically significant. This study emphasized the usefulness of the AI-assisted system for cases in which pathologists had difficulty determining the PD-L1 TPS.

Viable tumor cell density after neoadjuvant chemotherapy assessed using deep learning model reflects the prognosis of osteosarcoma.

Prognosis after neoadjuvant chemotherapy (NAC) for osteosarcoma is generally predicted using manual necrosis-rate assessments; however, necrosis rates obtained in these assessments are not reproducible and do not adequately reflect individual cell responses. We aimed to investigate whether viable tumor cell density assessed using a deep-learning model (DLM) reflects the prognosis of osteosarcoma. Seventy-one patients were included in this study. Initially, the DLM was trained to detect viable tumor cells, following which it calculated their density. Patients were stratified into high and low-viable tumor cell density groups based on DLM measurements, and survival analysis was performed to evaluate disease-specific survival and metastasis-free survival (DSS and MFS). The high viable tumor cell density group exhibited worse DSS (p = 0.023) and MFS (p = 0.033). DLM-evaluated viable density showed correct stratification of prognosis groups. Therefore, this evaluation method may enable precise stratification of the prognosis in osteosarcoma patients treated with NAC.

Cell Tracking in C. elegans with Cell Position Heatmap-Based Alignment and Pairwise Detection.

3D cell tracking in a living organism has a crucial role in live cell image analysis. Cell tracking in C. elegans has two difficulties. First, cell migration in a consecutive frame is large since they move their head during scanning. Second, cell detection is often inconsistent in consecutive frames due to touching cells and low-contrast images, and these inconsistent detections affect the tracking performance worse. In this paper, we propose a cell tracking method to address these issues, which has two main contributions. First, we introduce cell position heatmap-based non-rigid alignment with test-time fine-tuning, which can warp the detected points to near the positions at the next frame. Second, we propose a pairwise detection method, which uses the information of detection results at the previous frame for detecting cells at the current frame. The experimental results demonstrate the effectiveness of each module, and the proposed method achieved the best performance in comparison.

Artificial intelligence quantifying endoscopic severity of ulcerative colitis in gradation scale.

OBJECTIVES: Existing endoscopic scores for ulcerative colitis (UC) objectively categorize disease severity based on the presence or absence of endoscopic findings; therefore, it may not reflect the range of clinical severity within each category. However, inflammatory bowel disease (IBD) expert endoscopists categorize the severity and diagnose the overall impression of the degree of inflammation. This study aimed to develop an artificial intelligence (AI) system that can accurately represent the assessment of the endoscopic severity of UC by IBD expert endoscopists. METHODS: A ranking-convolutional neural network (ranking-CNN) was trained using comparative information on the UC severity of 13,826 pairs of endoscopic images created by IBD expert endoscopists. Using the trained ranking-CNN, the UC Endoscopic Gradation Scale (UCEGS) was used to express severity. Correlation coefficients were calculated to ensure that there were no inconsistencies in assessments of severity made using UCEGS diagnosed by the AI and the Mayo Endoscopic Subscore, and the correlation coefficients of the mean for test images assessed using UCEGS by four IBD expert endoscopists and the AI. RESULTS: Spearman's correlation coefficient between the UCEGS diagnosed by AI and Mayo Endoscopic Subscore was approximately 0.89. The correlation coefficients between IBD expert endoscopists and the AI of the evaluation results were all higher than 0.95 (P < 0.01). CONCLUSIONS: The AI developed here can diagnose UC severity endoscopically similar to IBD expert endoscopists.

Deep Learning for Predicting Effect of Neoadjuvant Therapies in Non-Small Cell Lung Carcinomas With Histologic Images.

Neoadjuvant therapies are used for locally advanced non-small cell lung carcinomas, whereby pathologists histologically evaluate the effect using resected specimens. Major pathological response (MPR) has recently been used for treatment evaluation and as an economical survival surrogate; however, interobserver variability and poor reproducibility are often noted. The aim of this study was to develop a deep learning (DL) model to predict MPR from hematoxylin and eosin-stained tissue images and to validate its utility for clinical use. We collected data on 125 primary non-small cell lung carcinoma cases that were resected after neoadjuvant therapy. The cases were randomly divided into 55 for training/validation and 70 for testing. A total of 261 hematoxylin and eosin-stained slides were obtained from the maximum tumor beds, and whole slide images were prepared. We used a multiscale patch model that can adaptively weight multiple convolutional neural networks trained with different field-of-view images. We performed 3-fold cross-validation to evaluate the model. During testing, we compared the percentages of viable tumor evaluated by annotator pathologists (reviewed data), those evaluated by nonannotator pathologists (primary data), and those predicted by the DL-based model using 2-class confusion matrices and receiver operating characteristic curves and performed a survival analysis between MPR-achieved and non-MPR cases. In cross-validation, accuracy and mean F1 score were 0.859 and 0.805, respectively. During testing, accuracy and mean F1 score with reviewed data and those with primary data were 0.986, 0.985, 0.943, and 0.943, respectively. The areas under the receiver operating characteristic curve with reviewed and primary data were 0.999 and 0.978, respectively. The disease-free survival of MPR-achieved cases with reviewed and primary data was significantly better than that of the non-MPR cases (P<.001 and P=.001), and that predicted by the DL-based model was almost identical (P=.005). The DL model may support pathologist evaluations and can offer accurate determinations of MPR in patients.

Analysis of optical absorption of photoaged human skin using a high-frequency illumination microscopy analysis system.

Skin is composed of different layers, including the stratum corneum, epidermal living layer and papillary and reticular dermis. Each has specific optical properties due to differences in their biological components. Alterations in the skin's cutaneous biological components resulting from photoaging caused by chronic exposure to UV light affect the deterioration of appearance associated with the skin's optical properties. Various methods for analysing cutaneous optical properties have been previously proposed, including mathematical models and computer simulations. However, these were insufficient to elucidate changes in each skin layer and comprehensively understand the skin's integrated optical properties. We focused on UV-induced yellowing of the facial skin. We evaluated site-specific optical absorption of human skin tissue sections to investigate the yellowish discoloration, which is suggested to be related to the photodamage process. The method includes our original technique of separating the transmitted and scattered light using high-frequency illumination microscopy, leading to microscopic analysis of the tissue's optical absorption in the regions of interest. In analysing the sun-exposed facial skin tissue sections, we successfully showed that dermal regions of aged skin have increased absorption at 450 nm, where yellowish colours are complemented. Furthermore, we confirmed that elastic fibres with observable histological disorder resulting from photodamage are a prominent source of high optical absorption. We detected changes in the skin's optical absorption associated with dermal degeneration resulting from photodamage using a novel optical microscopy technique. The results provide a base for the evaluation of optical property changes for both yellowing discoloration and other tissue disorders.

Deep learning-based automatic-bone-destruction-evaluation system using contextual information from other joints.

BACKGROUND: X-ray images are commonly used to assess the bone destruction of rheumatoid arthritis. The purpose of this study is to propose an automatic-bone-destruction-evaluation system fully utilizing deep neural networks (DNN). This system detects all target joints of the modified Sharp/van der Heijde score (SHS) from a hand X-ray image. It then classifies every target joint as intact (SHS = 0) or non-intact (SHS ≥ 1). METHODS: We used 226 hand X-ray images of 40 rheumatoid arthritis patients. As for detection, we used a DNN model called DeepLabCut. As for classification, we built four classification models that classify the detected joint as intact or non-intact. The first model classifies each joint independently, whereas the second model does it while comparing the same contralateral joint. The third model compares the same joint group (e.g., the proximal interphalangeal joints) of one hand and the fourth model compares the same joint group of both hands. We evaluated DeepLabCut's detection performance and classification models' performances. The classification models' performances were compared to three orthopedic surgeons. RESULTS: Detection rates for all the target joints were 98.0% and 97.3% for erosion and joint space narrowing (JSN). Among the four classification models, the model that compares the same contralateral joint showed the best F-measure (0.70, 0.81) and area under the curve of the precision-recall curve (PR-AUC) (0.73, 0.85) regarding erosion and JSN. As for erosion, the F-measure and PR-AUC of this model were better than the best of the orthopedic surgeons. CONCLUSIONS: The proposed system was useful. All the target joints were detected with high accuracy. The classification model that compared the same contralateral joint showed better performance than the orthopedic surgeons regarding erosion.

Effective pseudo-labeling based on heatmap for unsupervised domain adaptation in cell detection.

Cell detection is an important task in biomedical research. Recently, deep learning methods have made it possible to improve the performance of cell detection. However, a detection network trained with training data under a specific condition (source domain) may not work well on data under other conditions (target domains), which is called the domain shift problem. In particular, cells are cultured under different conditions depending on the purpose of the research. Characteristics, e.g., the shapes and density of the cells, change depending on the conditions, and such changes may cause domain shift problems. Here, we propose an unsupervised domain adaptation method for cell detection using a pseudo-cell-position heatmap, where the cell centroid is at the peak of a Gaussian distribution in the map and selective pseudo-labeling. In the prediction result for the target domain, even if the peak location is correct, the signal distribution around the peak often has a non-Gaussian shape. The pseudo-cell-position heatmap is thus re-generated using the peak positions in the predicted heatmap to have a clear Gaussian shape. Our method selects confident pseudo-cell-position heatmaps based on uncertainty and curriculum learning. We conducted numerous experiments showing that, compared with the existing methods, our method improved detection performance under different conditions.

Estimation of Wetness and Color from a Single Multispectral Image.

Recognizing wet surfaces and their degrees of wetness is essential for many computer vision applications. Surface wetness can inform us slippery spots on a road to autonomous vehicles, muddy areas of a trail to humanoid robots, and the freshness of groceries to us. The fact that surfaces darken when wet, i.e., monochromatic appearance change, has been modeled to recognize wet surfaces in the past. In this paper, we show that color change, particularly in its spectral behavior, carries rich information about surface wetness. We first derive an analytical spectral appearance model of wet surfaces that expresses the characteristic spectral sharpening due to multiple scattering and absorption in the surface. We present a novel method for estimating key parameters of this spectral appearance model, which enables the recovery of the original surface color and the degree of wetness from a single multispectral image. Applied to a multispectral image, the method estimates the spatial map of wetness together with the dry spectral distribution of the surface. To our knowledge, this is the first work to model and leverage the spectral characteristics of wet surfaces to decipher its appearance. We conduct comprehensive experimental validation with a number of wet real surfaces. The results demonstrate the accuracy of our model and the effectiveness of our method for surface wetness and color estimation.

Patch-Based Cervical Cancer Segmentation using Distance from Boundary of Tissue.

Pathological diagnosis is used for examining cancer in detail, and its automation is in demand. To automatically segment each cancer area, a patch-based approach is usually used since a Whole Slide Image (WSI) is huge. However, this approach loses the global information needed to distinguish between classes. In this paper, we utilized the Distance from the Boundary of tissue (DfB), which is global information that can be extracted from the original image. We experimentally applied our method to the three-class classification of cervical cancer, and found that it improved the total performance compared with the conventional method.

Unsupervised Body Hair Detection by Positive-Unlabeled Learning in Photoacoustic Image.

Photoacoustic (PA) imaging is a new imaging technology that can non-invasively visualize blood vessels and body hair in 3D. It is useful in cosmetic surgery for detecting body hair and computing metrics such as the number and thicknesses of hairs. Previous supervised body hair detection methods often do not work if the imaging conditions change from training data. We propose an unsupervised hair detection method. Hair samples were automatically extracted from unlabeled samples using prior knowledge about spatial structure. If hair (positive) samples and unlabeled samples are obtained, Positive Unlabeled (PU) learning becomes possible. PU methods can learn a binary classifier from positive samples and unlabeled samples. The advantage of the proposed method is that it can estimate an appropriate decision boundary in accordance with the distribution of the test data. Experimental results using real PA data demonstrate that the proposed approach effectively detects body hairs.

Weakly supervised cell instance segmentation under various conditions.

Cell instance segmentation is important in biomedical research. For living cell analysis, microscopy images are captured under various conditions (e.g., the type of microscopy and type of cell). Deep-learning-based methods can be used to perform instance segmentation if sufficient annotations of individual cell boundaries are prepared as training data. Generally, annotations are required for each condition, which is very time-consuming and labor-intensive. To reduce the annotation cost, we propose a weakly supervised cell instance segmentation method that can segment individual cell regions under various conditions by only using rough cell centroid positions as training data. This method dramatically reduces the annotation cost compared with the standard annotation method of supervised segmentation. We demonstrated the efficacy of our method on various cell images; it outperformed several of the conventional weakly-supervised methods on average. In addition, we demonstrated that our method can perform instance cell segmentation without any manual annotation by using pairs of phase contrast and fluorescence images in which cell nuclei are stained as training data.

Soft and self constrained clustering for group-based labeling.

When using deep neural networks in medical image classification tasks, it is mandatory to prepare a large-scale labeled image set, and this often requires significant effort by medical experts. One strategy to reduce the labeling cost is group-based labeling, where image samples are clustered and then a label is attached to each cluster. The efficiency of this strategy depends on the purity of the clusters. Constrained clustering is an effective way to improve the purity of the clusters if we can give appropriate must-links and cannot-links as constraints. However, for medical image clustering, the conventional constrained clustering methods encounter two issues. The first issue is that constraints are not always appropriate due to the gap between semantic and visual similarities. The second issue is that attaching constraints requires extra effort from medical experts. To deal with the first issue, we propose a novel soft-constrained clustering method, which has the ability to ignore inappropriate constraints. To deal with the second issue, we propose a self-constrained clustering method that utilizes prior knowledge about the target images to set the constraints automatically. Experiments with the endoscopic image datasets demonstrated that the proposed methods give clustering results with higher purity.

Light-sheet microscopy-based 3D single-cell tracking reveals a correlation between cell cycle and the start of endoderm cell internalization in early zebrafish development.

Controlling the initiation of cell migration plays a fundamental role in shaping the tissue during embryonic development. During gastrulation in zebrafish, some mesendoderm cells migrate inward to form the endoderm as the innermost germ layer along the yolk syncytial layer. However, how the initiation of inward migration is regulated is poorly understood. In this study, we performed light-sheet microscopy-based 3D single-cell tracking consisting of (a) whole-embryo time-lapse imaging with light-sheet microscopy and (b) three-dimensional single cell tracking in the zebrafish gastrula in which cells are marked with histone H2A-mCherry (nuclei) and the sox17:EGFP transgene (expressed in endoderm cells). We analyzed the correlation between the timing of cell internalization and cell division. Most cells that differentiated into endoderm cells began to internalize during the first half of the cell cycle, where the length of a cell cycle was defined by the period between two successive cell divisions. By contrast, the timing of other internalized cells was not correlated with a certain phase of the cell cycle. These results suggest the possibility that cell differentiation is associated with the relationship between cell cycle progression and the start of internalization. Moreover, the 3D single-cell tracking approach is useful for further investigating how cell migration is integrated with cell proliferation to shape tissues in zebrafish embryos.

Spatial-Temporal Mitosis Detection in Phase-Contrast Microscopy via Likelihood Map Estimation by 3DCNN.

Automated mitotic detection in time-lapse phase-contrast microscopy provides us much information for cell behavior analysis, and thus several mitosis detection methods have been proposed. However, these methods still have two problems; 1) they cannot detect multiple mitosis events when there are closely placed. 2) they do not consider the annotation gaps, which may occur since the appearances of mitosis cells are very similar before and after the annotated frame. In this paper, we propose a novel mitosis detection method that can detect multiple mitosis events in a candidate sequence and mitigate the human annotation gap via estimating spatial-temporal likelihood map by 3DCNN. In this training, the loss gradually decreases with the gap size between ground-truth and estimation. This mitigates the annotation gaps. Our method outperformed the compared methods in terms of F1-score using challenging dataset that contains the data under four different conditions. Code is publicly available in https://github.com/naivete5656/MDMLM.

Combined multiphoton imaging and biaxial tissue extension for quantitative analysis of geometric fiber organization in human reticular dermis.

The geometric organization of collagen fibers in human reticular dermis and its relationship to that of elastic fibers remain unclear. The tight packing and complex intertwining of dermal collagen fibers hinder accurate analysis of fiber orientation. We hypothesized that combined multiphoton microscopy and biaxial extension could overcome this issue. Continuous observation of fresh dermal sheets under biaxial extension revealed that the geometry of the elastic fiber network is maintained during expansion. Full-thickness human thigh skin samples were biaxially extended and cleared to visualize the entire reticular dermis. Throughout the dermis, collagen fibers straightened with increased inter-fiber spaces, making them more clearly identifiable after extension. The distribution of collagen fibers was evaluated with compilation of local orientation data. Two or three modes were confirmed in all superficial reticular layer samples. A high degree of local similarities in the direction of collagen and elastic fibers was observed. More than 80% of fibers had directional differences of ≤15°, regardless of layer. Understanding the geometric organization of fibers in the reticular dermis improves the understanding of mechanisms underlying the pliability of human skin. Combined multiphoton imaging and biaxial extension provides a research tool for studying the fibrous microarchitecture of the skin.

Digital artery deformation on movement of the proximal interphalangeal joint.

This study aimed to characterize in vivo human digital arteries in three-dimensions using photoacoustic tomography in order to understand the specific mechanism underlying arterial deformation associated with movement of the proximal interphalangeal joint. Three-dimensional morphological data were obtained on the radialis indicis artery (radial artery of the index finger) at different angles of the joint. The association between increased curvature of the deformation and the anatomical region was assessed. Characteristic morphological deformations in areas of major deformation were determined. The deformation of the artery was characterized by three consecutive curves in juxta-articular regions, which were particularly noticeable when the joint was flexed at an angle of ≥ 60°. The change in the curvature of the deformation during 30°-90° of flexion was lower in middle-aged individuals than in young individuals. Better understanding of the mechanism underlying deformation of the digital arteries may contribute to advancements in flap procedures and rehabilitation strategies after digital artery repair.

Endoscopic Image Clustering with Temporal Ordering Information Based on Dynamic Programming.

In this paper, we propose a clustering method with temporal ordering information for endoscopic image sequences. It is difficult to collect a sufficient amount of endoscopic image datasets to train machine learning techniques by manual labeling. The clustering of endoscopic images leads to group-based labeling, which is useful for reducing the cost of dataset construction. Therefore, in this paper, we propose a clustering method where the property of endoscopic image sequences is fully utilized. For the proposed method, a deep neural network was used to extract features from endoscopic images, and clustering with temporal ordering information was solved by dynamic programming. In the experiments, we clustered the esophagogastroduodenoscopy images. From the results, we confirmed that the performance was improved by using the sequential property.