Improving Image Segmentation with Contextual and Structural Similarity.

Deep learning models for medical image segmentation are usually trained with voxel-wise losses, e.g., cross-entropy loss, focusing on unary supervision without considering inter-voxel relationships. This oversight potentially leads to semantically inconsistent predictions. Here, we propose a contextual similarity loss (CSL) and a structural similarity loss (SSL) to explicitly and efficiently incorporate inter-voxel relationships for improved performance. The CSL promotes consistency in predicted object categories for each image sub-region compared to ground truth. The SSL enforces compatibility between the predictions of voxel pairs by computing pair-wise distances between them, ensuring that voxels of the same class are close together whereas those from different classes are separated by a wide margin in the distribution space. The effectiveness of the CSL and SSL is evaluated using a clinical cone-beam computed tomography (CBCT) dataset of patients with various craniomaxillofacial (CMF) deformities and a public pancreas dataset. Experimental results show that the CSL and SSL outperform state-of-the-art regional loss functions in preserving segmentation semantics.

Correspondence attention for facial appearance simulation.

In orthognathic surgical planning for patients with jaw deformities, it is crucial to accurately simulate the changes in facial appearance that follow the bony movement. Compared with the traditional biomechanics-based methods like the finite-element method (FEM), which are both labor-intensive and computationally inefficient, deep learning-based methods offer an efficient and robust modeling alternative. However, current methods do not account for the physical relationship between facial soft tissue and bony structure, causing them to fall short in accuracy compared to FEM. In this work, we propose an Attentive Correspondence assisted Movement Transformation network (ACMT-Net) to predict facial changes by correlating facial soft tissue changes with bony movement through a point-to-point attentive correspondence matrix. To ensure efficient training, we also introduce a contrastive loss for self-supervised pre-training of the ACMT-Net with a k-Nearest Neighbors (k-NN) based clustering. Experimental results on patients with jaw deformities show that our proposed solution can achieve significantly improved computational efficiency over the state-of-the-art FEM-based method with comparable facial change prediction accuracy.

Accuracy of Surgical Outcome Using Computer-Aided Surgical Simulation in Fronto-Orbital Advancement for Craniosynostosis: A Pilot Study.

BACKGROUND AND OBJECTIVE: Computer-aided surgical simulation (CASS) can be used to virtually plan ideal outcomes of craniosynostosis surgery. Our purpose was to create a workflow analyzing the accuracy of surgical outcomes relative to virtually planned fronto-orbital advancement (FOA). METHODS: Patients who underwent FOA using CASS between October 1, 2017, and February 28, 2022, at our center and had postoperative computed tomography within 6 months of surgery were included. Virtual 3-dimensional (3D) models were created and coregistered using each patient's preoperative and postoperative computed tomography data. Three points on each bony segment were used to define the object in 3D space. Each planned bony segment was manipulated to match the actual postoperative outcome. The change in position of the 3D object was measured in translational (X, Y, Z) and rotational (roll, pitch, yaw) aspects to represent differences between planned and actual postoperative positions. The difference in the translational position of several bony landmarks was also recorded. Wilcoxon signed-rank tests were performed to measure significance of these differences from the ideal value of 0, which would indicate no difference between preoperative plan and postoperative outcome. RESULTS: Data for 63 bony segments were analyzed from 8 patients who met the inclusion criteria. Median differences between planned and actual outcomes of the segment groups ranged from -0.3 to -1.3 mm in the X plane; 1.4 to 5.6 mm in the Y plane; 0.9 to 2.7 mm in the Z plane; -1.2° to -4.5° in pitch; -0.1° to 1.0° in roll; and -2.8° to 1.0° in yaw. No significant difference from 0 was found in 21 of 24 segment region/side combinations. Translational differences of bony landmarks ranged from -2.7 to 3.6 mm. CONCLUSION: A high degree of accuracy was observed relative to the CASS plan. Virtual analysis of surgical accuracy in FOA using CASS was feasible.

Machine Learning Effectively Diagnoses Mandibular Deformity Using Three-Dimensional Landmarks.

BACKGROUND: Jaw deformity diagnosis requires objective tests. Current methods, like cephalometry, have limitations. However, recent studies have shown that machine learning can diagnose jaw deformities in two dimensions. Therefore, we hypothesized that a multilayer perceptron (MLP) could accurately diagnose jaw deformities in three dimensions (3D). PURPOSE: Examine the hypothesis by focusing on anomalous mandibular position. We aimed to: (1) create a machine learning model to diagnose mandibular retrognathism and prognathism; and (2) compare its performance with traditional cephalometric methods. STUDY DESIGN, SETTING, SAMPLE: An in-silico experiment on deidentified retrospective data. The study was conducted at the Houston Methodist Research Institute and Rensselaer Polytechnic Institute. Included were patient records with jaw deformities and preoperative 3D facial models. Patients with significant jaw asymmetry were excluded. PREDICTOR VARIABLES: The tests used to diagnose mandibular anteroposterior position are: (1) SNB angle; (2) facial angle; (3) mandibular unit length (MdUL); and (4) MLP model. MAIN OUTCOME VARIABLE: The resultant diagnoses: normal, prognathic, or retrognathic. COVARIATES: None. ANALYSES: A senior surgeon labeled the patients' mandibles as prognathic, normal, or retrognathic, creating a gold standard. Scientists at Rensselaer Polytechnic Institute developed an MLP model to diagnose mandibular prognathism and retrognathism using the 3D coordinates of 50 landmarks. The performance of the MLP model was compared with three traditional cephalometric measurements: (1) SNB, (2) facial angle, and (3) MdUL. The primary metric used to assess the performance was diagnostic accuracy. McNemar's exact test tested the difference between traditional cephalometric measurement and MLP. Cohen's Kappa measured inter-rater agreement between each method and the gold standard. RESULTS: The sample included 101 patients. The diagnostic accuracy of SNB, facial angle, MdUL, and MLP were 74.3, 74.3, 75.3, and 85.2%, respectively. McNemar's test shows that our MLP performs significantly better than the SNB (P = .027), facial angle (P = .019), and MdUL (P = .031). The agreement between the traditional cephalometric measurements and the surgeon's diagnosis was fair. In contrast, the agreement between the MLP and the surgeon was moderate. CONCLUSION AND RELEVANCE: The performance of the MLP is significantly better than that of the traditional cephalometric measurements.

Federated Multi-Organ Segmentation With Inconsistent Labels.

Federated learning is an emerging paradigm allowing large-scale decentralized learning without sharing data across different data owners, which helps address the concern of data privacy in medical image analysis. However, the requirement for label consistency across clients by the existing methods largely narrows its application scope. In practice, each clinical site may only annotate certain organs of interest with partial or no overlap with other sites. Incorporating such partially labeled data into a unified federation is an unexplored problem with clinical significance and urgency. This work tackles the challenge by using a novel federated multi-encoding U-Net (Fed-MENU) method for multi-organ segmentation. In our method, a multi-encoding U-Net (MENU-Net) is proposed to extract organ-specific features through different encoding sub-networks. Each sub-network can be seen as an expert of a specific organ and trained for that client. Moreover, to encourage the organ-specific features extracted by different sub-networks to be informative and distinctive, we regularize the training of the MENU-Net by designing an auxiliary generic decoder (AGD). Extensive experiments on six public abdominal CT datasets show that our Fed-MENU method can effectively obtain a federated learning model using the partially labeled datasets with superior performance to other models trained by either localized or centralized learning methods. Source code is publicly available at https://github.com/DIAL-RPI/Fed-MENU.

Deep learning for biomechanical modeling of facial tissue deformation in orthognathic surgical planning.

PURPOSE: Orthognathic surgery requires an accurate surgical plan of how bony segments are moved and how the face passively responds to the bony movement. Currently, finite element method (FEM) is the standard for predicting facial deformation. Deep learning models have recently been used to approximate FEM because of their faster simulation speed. However, current solutions are not compatible with detailed facial meshes and often do not explicitly provide the network with known boundary type information. Therefore, the purpose of this proof-of-concept study is to develop a biomechanics-informed deep neural network that accepts point cloud data and explicit boundary types as inputs to the network for fast prediction of soft-tissue deformation. METHODS: A deep learning network was developed based on the PointNet++ architecture. The network accepts the starting facial mesh, input displacement, and explicit boundary type information and predicts the final facial mesh deformation. RESULTS: We trained and tested our deep learning model on datasets created from FEM simulations of facial meshes. Our model achieved a mean error between 0.159 and 0.642 mm on five subjects. Including explicit boundary types had mixed results, improving performance in simulations with large deformations but decreasing performance in simulations with small deformations. These results suggest that including explicit boundary types may not be necessary to improve network performance. CONCLUSION: Our deep learning method can approximate FEM for facial change prediction in orthognathic surgical planning by accepting geometrically detailed meshes and explicit boundary types while significantly reducing simulation time.

Midsagittal Plane First: Building a Strong Facial Reference Frame for Computer-Aided Surgical Simulation.

PURPOSE: A facial reference frame is a 3-dimensional Cartesian coordinate system that includes 3 perpendicular planes: midsagittal, axial, and coronal. The order in which one defines the planes matters. The purposes of this study are to determine the following: 1) what sequence (axial-midsagittal-coronal vs midsagittal-axial-coronal) produced more appropriate reference frames and 2) whether orbital or auricular dystopia influenced the outcomes. METHODS: This study is an ambispective cross-sectional study. Fifty-four subjects with facial asymmetry were included. The facial reference frames of each subject (outcome variable) were constructed using 2 methods (independent variable): axial plane first and midsagittal plane first. Two board-certified orthodontists together blindly evaluated the results using a 3-point categorical scale based on their careful inspection and expert intuition. The covariant for stratification was the existence of orbital or auricular dystopia. Finally, Wilcoxon signed rank tests were performed. RESULTS: The facial reference frames defined by the midsagittal plane first method was statistically significantly different from ones defined by the axial plane first method (P = .001). Using the midsagittal plane first method, the reference frames were more appropriately defined in 22 (40.7%) subjects, equivalent in 26 (48.1%) and less appropriately defined in 6 (11.1%). After stratified by orbital or auricular dystopia, the results also showed that the reference frame computed using midsagittal plane first method was statistically significantly more appropriate in both subject groups regardless of the existence of orbital or auricular dystopia (27 with orbital or auricular dystopia and 27 without, both P < .05). CONCLUSIONS: The midsagittal plane first sequence improves the facial reference frames compared with the traditional axial plane first approach. However, regardless of the sequence used, clinicians need to judge the correctness of the reference frame before diagnosis or surgical planning.

Deep Simulation of Facial Appearance Changes Following Craniomaxillofacial Bony Movements in Orthognathic Surgical Planning.

Facial appearance changes with the movements of bony segments in orthognathic surgery of patients with craniomaxillofacial (CMF) deformities. Conventional bio-mechanical methods, such as finite element modeling (FEM), for simulating such changes, are labor intensive and computationally expensive, preventing them from being used in clinical settings. To overcome these limitations, we propose a deep learning framework to predict post-operative facial changes. Specifically, FC-Net, a facial appearance change simulation network, is developed to predict the point displacement vectors associated with a facial point cloud. FC-Net learns the point displacements of a pre-operative facial point cloud from the bony movement vectors between pre-operative and simulated post-operative bony models. FC-Net is a weakly-supervised point displacement network trained using paired data with strict point-to-point correspondence. To preserve the topology of the facial model during point transform, we employ a local-point-transform loss to constrain the local movements of points. Experimental results on real patient data reveal that the proposed framework can predict post-operative facial appearance changes remarkably faster than a state-of-the-art FEM method with comparable prediction accuracy.

DLLNet: An Attention-Based Deep Learning Method for Dental Landmark Localization on High-Resolution 3D Digital Dental Models.

Dental landmark localization is a fundamental step to analyzing dental models in the planning of orthodontic or orthognathic surgery. However, current clinical practices require clinicians to manually digitize more than 60 landmarks on 3D dental models. Automatic methods to detect landmarks can release clinicians from the tedious labor of manual annotation and improve localization accuracy. Most existing landmark detection methods fail to capture local geometric contexts, causing large errors and misdetections. We propose an end-to-end learning framework to automatically localize 68 landmarks on high-resolution dental surfaces. Our network hierarchically extracts multi-scale local contextual features along two paths: a landmark localization path and a landmark area-of-interest segmentation path. Higher-level features are learned by combining local-to-global features from the two paths by feature fusion to predict the landmark heatmap and the landmark area segmentation map. An attention mechanism is then applied to the two maps to refine the landmark position. We evaluated our framework on a real-patient dataset consisting of 77 high-resolution dental surfaces. Our approach achieves an average localization error of 0.42 mm, significantly outperforming related start-of-the-art methods.

Fast and Accurate Craniomaxillofacial Landmark Detection via 3D Faster R-CNN.

Automatic craniomaxillofacial (CMF) landmark localization from cone-beam computed tomography (CBCT) images is challenging, considering that 1) the number of landmarks in the images may change due to varying deformities and traumatic defects, and 2) the CBCT images used in clinical practice are typically large. In this paper, we propose a two-stage, coarse-to-fine deep learning method to tackle these challenges with both speed and accuracy in mind. Specifically, we first use a 3D faster R-CNN to roughly locate landmarks in down-sampled CBCT images that have varying numbers of landmarks. By converting the landmark point detection problem to a generic object detection problem, our 3D faster R-CNN is formulated to detect virtual, fixed-size objects in small boxes with centers indicating the approximate locations of the landmarks. Based on the rough landmark locations, we then crop 3D patches from the high-resolution images and send them to a multi-scale UNet for the regression of heatmaps, from which the refined landmark locations are finally derived. We evaluated the proposed approach by detecting up to 18 landmarks on a real clinical dataset of CMF CBCT images with various conditions. Experiments show that our approach achieves state-of-the-art accuracy of 0.89 ± 0.64mm in an average time of 26.2 seconds per volume.

A novel incremental simulation of facial changes following orthognathic surgery using FEM with realistic lip sliding effect.

Accurate prediction of facial soft-tissue changes following orthognathic surgery is crucial for surgical outcome improvement. We developed a novel incremental simulation approach using finite element method (FEM) with a realistic lip sliding effect to improve the prediction accuracy in the lip region. First, a lip-detailed mesh is generated based on accurately digitized lip surface points. Second, an improved facial soft-tissue change simulation method is developed by applying a lip sliding effect along with the mucosa sliding effect. Finally, the orthognathic surgery initiated soft-tissue change is simulated incrementally to facilitate a natural transition of the facial change and improve the effectiveness of the sliding effects. Our method was quantitatively validated using 35 retrospective clinical data sets by comparing it to the traditional FEM simulation method and the FEM simulation method with mucosa sliding effect only. The surface deviation error of our method showed significant improvement in the upper and lower lips over the other two prior methods. In addition, the evaluation results using our lip-shape analysis, which reflects clinician's qualitative evaluation, also proved significant improvement of the lip prediction accuracy of our method for the lower lip and both upper and lower lips as a whole compared to the other two methods. In conclusion, the prediction accuracy in the clinically critical region, i.e., the lips, significantly improved after applying incremental simulation with realistic lip sliding effect compared with the FEM simulation methods without the lip sliding effect.

Multi-task Dynamic Transformer Network for Concurrent Bone Segmentation and Large-Scale Landmark Localization with Dental CBCT.

Accurate bone segmentation and anatomical landmark localization are essential tasks in computer-aided surgical simulation for patients with craniomaxillofacial (CMF) deformities. To leverage the complementarity between the two tasks, we propose an efficient end-to-end deep network, i.e., multi-task dynamic transformer network (DTNet), to concurrently segment CMF bones and localize large-scale landmarks in one-pass from large volumes of cone-beam computed tomography (CBCT) data. Our DTNet was evaluated quantitatively using CBCTs of patients with CMF deformities. The results demonstrated that our method outperforms the other state-of-the-art methods in both tasks of the bony segmentation and the landmark digitization. Our DTNet features three main technical contributions. First, a collaborative two-branch architecture is designed to efficiently capture both fine-grained image details and complete global context for high-resolution volume-to-volume prediction. Second, leveraging anatomical dependencies between landmarks, regionalized dynamic learners (RDLs) are designed in the concept of "learns to learn" to jointly regress large-scale 3D heatmaps of all landmarks under limited computational costs. Third, adaptive transformer modules (ATMs) are designed for the flexible learning of task-specific feature embedding from common feature bases.

Craniomaxillofacial Bony Structures Segmentation from MRI with Deep-Supervision Adversarial Learning.

Automatic segmentation of medical images finds abundant applications in clinical studies. Computed Tomography (CT) imaging plays a critical role in diagnostic and surgical planning of craniomaxillofacial (CMF) surgeries as it shows clear bony structures. However, CT imaging poses radiation risks for the subjects being scanned. Alternatively, Magnetic Resonance Imaging (MRI) is considered to be safe and provides good visualization of the soft tissues, but the bony structures appear invisible from MRI. Therefore, the segmentation of bony structures from MRI is quite challenging. In this paper, we propose a cascaded generative adversarial network with deep-supervision discriminator (Deep-supGAN) for automatic bony structures segmentation. The first block in this architecture is used to generate a high-quality CT image from an MRI, and the second block is used to segment bony structures from MRI and the generated CT image. Different from traditional discriminators, the deep-supervision discriminator distinguishes the generated CT from the ground-truth at different levels of feature maps. For segmentation, the loss is not only concentrated on the voxel level but also on the higher abstract perceptual levels. Experimental results show that the proposed method generates CT images with clearer structural details and also segments the bony structures more accurately compared with the state-of-the-art methods.