Multimodal learning on graphs for disease relation extraction.

Disease knowledge graphs have emerged as a powerful tool for artificial intelligence to connect, organize, and access diverse information about diseases. Relations between disease concepts are often distributed across multiple datasets, including unstructured plain text datasets and incomplete disease knowledge graphs. Extracting disease relations from multimodal data sources is thus crucial for constructing accurate and comprehensive disease knowledge graphs. We introduce REMAP, a multimodal approach for disease relation extraction. The REMAP machine learning approach jointly embeds a partial, incomplete knowledge graph and a medical language dataset into a compact latent vector space, aligning the multimodal embeddings for optimal disease relation extraction. Additionally, REMAP utilizes a decoupled model structure to enable inference in single-modal data, which can be applied under missing modality scenarios. We apply the REMAP approach to a disease knowledge graph with 96,913 relations and a text dataset of 1.24 million sentences. On a dataset annotated by human experts, REMAP improves language-based disease relation extraction by 10.0% (accuracy) and 17.2% (F1-score) by fusing disease knowledge graphs with language information. Furthermore, REMAP leverages text information to recommend new relationships in the knowledge graph, outperforming graph-based methods by 8.4% (accuracy) and 10.4% (F1-score). REMAP is a flexible multimodal approach for extracting disease relations by fusing structured knowledge and language information. This approach provides a powerful model to easily find, access, and evaluate relations between disease concepts.

A novel Anti-ROS osteoblast-specific delivery system for ankylosing spondylitis treatment via suppression of both inflammation and pathological new bone formation.

Ankylosing spondylitis (AS) is a common rheumatic disorder distinguished by chronic inflammation and heterotopic ossification at local entheses sites. Currently available medications, including nonsteroidal anti-inflammatory drugs (NSAIDs), disease-modifying anti-rheumatic drugs (DMARDs) and TNF inhibitors, are limited by side effects, high costs and unclear inhibitory effects on heterotopic ossification. Herein, we developed manganese ferrite nanoparticles modified by the aptamer CH6 (CH6-MF NPs) that can efficiently scavenge ROS and actively deliver siRNA into hMSCs and osteoblasts in vivo for effective AS treatment. CH6-MF NPs loaded with BMP2 siRNA (CH6-MF-Si NPs) effectively suppressed abnormal osteogenic differentiation under inflammatory conditions in vitro. During their circulation and passive accumulation in inflamed joints in the Zap70mut mouse model, CH6-MF-Si NPs attenuated local inflammation and rescued heterotopic ossification in the entheses. Thus, CH6-MF NPs may be an effective inflammation reliever and osteoblast-specific delivery system, and CH6-MF-Si NPs have potential for the dual treatment of chronic inflammation and heterotopic ossification in AS.

Generate Analysis-Ready Data for Real-world Evidence: Tutorial for Harnessing Electronic Health Records With Advanced Informatic Technologies.

Although randomized controlled trials (RCTs) are the gold standard for establishing the efficacy and safety of a medical treatment, real-world evidence (RWE) generated from real-world data has been vital in postapproval monitoring and is being promoted for the regulatory process of experimental therapies. An emerging source of real-world data is electronic health records (EHRs), which contain detailed information on patient care in both structured (eg, diagnosis codes) and unstructured (eg, clinical notes and images) forms. Despite the granularity of the data available in EHRs, the critical variables required to reliably assess the relationship between a treatment and clinical outcome are challenging to extract. To address this fundamental challenge and accelerate the reliable use of EHRs for RWE, we introduce an integrated data curation and modeling pipeline consisting of 4 modules that leverage recent advances in natural language processing, computational phenotyping, and causal modeling techniques with noisy data. Module 1 consists of techniques for data harmonization. We use natural language processing to recognize clinical variables from RCT design documents and map the extracted variables to EHR features with description matching and knowledge networks. Module 2 then develops techniques for cohort construction using advanced phenotyping algorithms to both identify patients with diseases of interest and define the treatment arms. Module 3 introduces methods for variable curation, including a list of existing tools to extract baseline variables from different sources (eg, codified, free text, and medical imaging) and end points of various types (eg, death, binary, temporal, and numerical). Finally, module 4 presents validation and robust modeling methods, and we propose a strategy to create gold-standard labels for EHR variables of interest to validate data curation quality and perform subsequent causal modeling for RWE. In addition to the workflow proposed in our pipeline, we also develop a reporting guideline for RWE that covers the necessary information to facilitate transparent reporting and reproducibility of results. Moreover, our pipeline is highly data driven, enhancing study data with a rich variety of publicly available information and knowledge sources. We also showcase our pipeline and provide guidance on the deployment of relevant tools by revisiting the emulation of the Clinical Outcomes of Surgical Therapy Study Group Trial on laparoscopy-assisted colectomy versus open colectomy in patients with early-stage colon cancer. We also draw on existing literature on EHR emulation of RCTs together with our own studies with the Mass General Brigham EHR.

Building a trustworthy AI differential diagnosis application for Crohn's disease and intestinal tuberculosis.

BACKGROUND: Differentiating between Crohn's disease (CD) and intestinal tuberculosis (ITB) with endoscopy is challenging. We aim to perform more accurate endoscopic diagnosis between CD and ITB by building a trustworthy AI differential diagnosis application. METHODS: A total of 1271 electronic health record (EHR) patients who had undergone colonoscopies at Peking Union Medical College Hospital (PUMCH) and were clinically diagnosed with CD (n = 875) or ITB (n = 396) were used in this study. We build a workflow to make diagnoses with EHRs and mine differential diagnosis features; this involves finetuning the pretrained language models, distilling them into a light and efficient TextCNN model, interpreting the neural network and selecting differential attribution features, and then adopting manual feature checking and carrying out debias training. RESULTS: The accuracy of debiased TextCNN on differential diagnosis between CD and ITB is 0.83 (CR F1: 0.87, ITB F1: 0.77), which is the best among the baselines. On the noisy validation set, its accuracy was 0.70 (CR F1: 0.87, ITB: 0.69), which was significantly higher than that of models without debias. We also find that the debiased model more easily mines the diagnostically significant features. The debiased TextCNN unearthed 39 diagnostic features in the form of phrases, 17 of which were key diagnostic features recognized by the guidelines. CONCLUSION: We build a trustworthy AI differential diagnosis application for differentiating between CD and ITB focusing on accuracy, interpretability and robustness. The classifiers perform well, and the features which had statistical significance were in agreement with clinical guidelines.

Automatic literature screening using the PAJO deep-learning model for clinical practice guidelines.

BACKGROUND: Clinical practice guidelines (CPGs) are designed to assist doctors in clinical decision making. High-quality research articles are important for the development of good CPGs. Commonly used manual screening processes are time-consuming and labor-intensive. Artificial intelligence (AI)-based techniques have been widely used to analyze unstructured data, including texts and images. Currently, there are no effective/efficient AI-based systems for screening literature. Therefore, developing an effective method for automatic literature screening can provide significant advantages. METHODS: Using advanced AI techniques, we propose the Paper title, Abstract, and Journal (PAJO) model, which treats article screening as a classification problem. For training, articles appearing in the current CPGs are treated as positive samples. The others are treated as negative samples. Then, the features of the texts (e.g., titles and abstracts) and journal characteristics are fully utilized by the PAJO model using the pretrained bidirectional-encoder-representations-from-transformers (BERT) model. The resulting text and journal encoders, along with the attention mechanism, are integrated in the PAJO model to complete the task. RESULTS: We collected 89,940 articles from PubMed to construct a dataset related to neck pain. Extensive experiments show that the PAJO model surpasses the state-of-the-art baseline by 1.91% (F1 score) and 2.25% (area under the receiver operating characteristic curve). Its prediction performance was also evaluated with respect to subject-matter experts, proving that PAJO can successfully screen high-quality articles. CONCLUSIONS: The PAJO model provides an effective solution for automatic literature screening. It can screen high-quality articles on neck pain and significantly improve the efficiency of CPG development. The methodology of PAJO can also be easily extended to other diseases for literature screening.

Weakly Semi-supervised phenotyping using Electronic Health records.

OBJECTIVE: Electronic Health Record (EHR) based phenotyping is a crucial yet challenging problem in the biomedical field. Though clinicians typically determine patient-level diagnoses via manual chart review, the sheer volume and heterogeneity of EHR data renders such tasks challenging, time-consuming, and prohibitively expensive, thus leading to a scarcity of clinical annotations in EHRs. Weakly supervised learning algorithms have been successfully applied to various EHR phenotyping problems, due to their ability to leverage information from large quantities of unlabeled samples to better inform predictions based on a far smaller number of patients. However, most weakly supervised methods are subject to the challenge to choose the right cutoff value to generate an optimal classifier. Furthermore, since they only utilize the most informative features (i.e., main ICD and NLP counts) they may fail for episodic phenotypes that cannot be consistently detected via ICD and NLP data. In this paper, we propose a label-efficient, weakly semi-supervised deep learning algorithm for EHR phenotyping (WSS-DL), which overcomes the limitations above. MATERIALS AND METHODS: WSS-DL classifies patient-level disease status through a series of learning stages: 1) generating silver standard labels, 2) deriving enhanced-silver-standard labels by fitting a weakly supervised deep learning model to data with silver standard labels as outcomes and high dimensional EHR features as input, and 3) obtaining the final prediction score and classifier by fitting a supervised learning model to data with a minimal number of gold standard labels as the outcome, and the enhanced-silver-standard labels and a minimal set of most informative EHR features as input. To assess the generalizability of WSS-DL across different phenotypes and medical institutions, we apply WSS-DL to classify a total of 17 diseases, including both acute and chronic conditions, using EHR data from three healthcare systems. Additionally, we determine the minimum quantity of training labels required by WSS-DL to outperform existing supervised and semi-supervised phenotyping methods. RESULTS: The proposed method, in combining the strengths of deep learning and weakly semi-supervised learning, successfully leverages the crucial phenotyping information contained in EHR features from unlabeled samples. Indeed, the deep learning model's ability to handle high-dimensional EHR features allows it to generate strong phenotype status predictions from silver standard labels. These predictions, in turn, provide highly effective features in the final logistic regression stage, leading to high phenotyping accuracy in notably small subsets of labeled data (e.g. n = 40 labeled samples). CONCLUSION: Our method's high performance in EHR datasets with very small numbers of labels indicates its potential value in aiding doctors to diagnose rare diseases as well as conditions susceptible to misdiagnosis.

Automatic text classification of actionable radiology reports of tinnitus patients using bidirectional encoder representations from transformer (BERT) and in-domain pre-training (IDPT).

BACKGROUND: Given the increasing number of people suffering from tinnitus, the accurate categorization of patients with actionable reports is attractive in assisting clinical decision making. However, this process requires experienced physicians and significant human labor. Natural language processing (NLP) has shown great potential in big data analytics of medical texts; yet, its application to domain-specific analysis of radiology reports is limited. OBJECTIVE: The aim of this study is to propose a novel approach in classifying actionable radiology reports of tinnitus patients using bidirectional encoder representations from transformer BERT-based models and evaluate the benefits of in domain pre-training (IDPT) along with a sequence adaptation strategy. METHODS: A total of 5864 temporal bone computed tomography(CT) reports are labeled by two experienced radiologists as follows: (1) normal findings without notable lesions; (2) notable lesions but uncorrelated to tinnitus; and (3) at least one lesion considered as potential cause of tinnitus. We then constructed a framework consisting of deep learning (DL) neural networks and self-supervised BERT models. A tinnitus domain-specific corpus is used to pre-train the BERT model to further improve its embedding weights. In addition, we conducted an experiment to evaluate multiple groups of max sequence length settings in BERT to reduce the excessive quantity of calculations. After a comprehensive comparison of all metrics, we determined the most promising approach through the performance comparison of F1-scores and AUC values. RESULTS: In the first experiment, the BERT finetune model achieved a more promising result (AUC-0.868, F1-0.760) compared with that of the Word2Vec-based models(AUC-0.767, F1-0.733) on validation data. In the second experiment, the BERT in-domain pre-training model (AUC-0.948, F1-0.841) performed significantly better than the BERT based model(AUC-0.868, F1-0.760). Additionally, in the variants of BERT fine-tuning models, Mengzi achieved the highest AUC of 0.878 (F1-0.764). Finally, we found that the BERT max-sequence-length of 128 tokens achieved an AUC of 0.866 (F1-0.736), which is almost equal to the BERT max-sequence-length of 512 tokens (AUC-0.868,F1-0.760). CONCLUSION: In conclusion, we developed a reliable BERT-based framework for tinnitus diagnosis from Chinese radiology reports, along with a sequence adaptation strategy to reduce computational resources while maintaining accuracy. The findings could provide a reference for NLP development in Chinese radiology reports.

[Safety of two ventilator weaning strategies after high-frequency oscillatory ventilation in preterm infants with respiratory distress syndrome: a prospective randomized controlled trial].

OBJECTIVE: To study the safety of two ventilator weaning strategies after high-frequency oscillatory ventilation (HFOV) for the treatment of neonatal respiratory distress syndrome (NRDS) in preterm infants. METHODS: A prospective randomized controlled trial was conducted for 101 preterm infants with NRDS, with a gestational age of ≤32+6 weeks or a birth weight of ≤1 500 g, who were admitted to the neonatal intensive care unit of Xiamen Maternal and Child Health Hospital from January 1, 2019 to June 30, 2020. The infants underwent HFOV as the preferred treatment. The infants were randomly divided into an observation group (50 infants with direct weaning from HFOV) and a control group (51 infants with weaning after HFOV was switched to conventional mechanical ventilation). The two groups were compared in terms of failure rate of ventilator weaning within 72 hours, changes in blood gas parameters at 2 hours before weaning and at 2 and 24 hours after weaning, respiratory support therapy, incidence rates of complications, and outcome at discharge. RESULTS: There was no significant difference in the failure rate of ventilator weaning within 72 hours (8% vs 14%, P > 0.05). The observation group had a significantly shorter duration of mechanical ventilation than the control group [(64±39) hours vs (88±69) hours, P < 0.05]. There were no significant differences between the two groups in the duration of mechanical ventilation, total oxygen supply time, blood gas parameters before and after ventilator weaning, incidence rates of complications, and outcome at discharge (P > 0.05). CONCLUSIONS: For preterm infants with NRDS, the strategy of weaning directly from HFOV is safe and reliable and can reduce the duration of invasive mechanical ventilation, and therefore, it holds promise for clinical application.

Long-distance disorder-disorder relation extraction with bootstrapped noisy data.

OBJECTIVE: Artificial intelligence in healthcare increasingly relies on relations in knowledge graphs for algorithm development. However, many important relations are not well covered in existing knowledge graphs. We aim to develop a novel long-distance relation extraction algorithm that leverages the article section structure and is trained with bootstrapped noisy data to identify important relations for diagnosis, including may cause, may be caused by, and differential diagnosis. METHODS: Known relations were extracted from semistructured web pages and a relational database and were paired with sentences containing corresponding medical concepts to form training data. The sentence form was extended to allow one concept to be in the title. An attention mechanism was applied to reduce the effect of noisily labeled sentences. Section structure embedding was added to provide additional context for relation expressions. Graph information was further incorporated into the model to differentiate the target relations whose expressions were often similar and interwoven. RESULTS: The extended sentence form allowed 1.75 times as many relations and 2.17 times as many sentences to be found compared to the conventional form. The various components of the proposed model all added to the accuracy. Overall, the positive sample accuracy of the proposed model was 9 percentage points higher than baseline deep learning models and 13 percentage points higher than naïve Bayes and support vector machines. CONCLUSION: Our bootstrap data preparation method and the extended sentence form could form a large training dataset to enable algorithm development and data mining efforts. Section structure embedding and graph information significantly increased prediction accuracy.

Can natural language processing help differentiate inflammatory intestinal diseases in China? Models applying random forest and convolutional neural network approaches.

BACKGROUND: Differentiating between ulcerative colitis (UC), Crohn's disease (CD) and intestinal tuberculosis (ITB) using endoscopy is challenging. We aimed to realize automatic differential diagnosis among these diseases through machine learning algorithms. METHODS: A total of 6399 consecutive patients (5128 UC, 875 CD and 396 ITB) who had undergone colonoscopy examinations in the Peking Union Medical College Hospital from January 2008 to November 2018 were enrolled. The input was the description of the endoscopic image in the form of free text. Word segmentation and key word filtering were conducted as data preprocessing. Random forest (RF) and convolutional neural network (CNN) approaches were applied to different disease entities. Three two-class classifiers (UC and CD, UC and ITB, and CD and ITB) and a three-class classifier (UC, CD and ITB) were built. RESULTS: The classifiers built in this research performed well, and the CNN had better performance in general. The RF sensitivities/specificities of UC-CD, UC-ITB, and CD-ITB were 0.89/0.84, 0.83/0.82, and 0.72/0.77, respectively, while the values for the CNN of CD-ITB were 0.90/0.77. The precisions/recalls of UC-CD-ITB when employing RF were 0.97/0.97, 0.65/0.53, and 0.68/0.76, respectively, and when employing the CNN were 0.99/0.97, 0.87/0.83, and 0.52/0.81, respectively. CONCLUSIONS: Classifiers built by RF and CNN approaches had excellent performance when classifying UC with CD or ITB. For the differentiation of CD and ITB, high specificity and sensitivity were achieved as well. Artificial intelligence through machine learning is very promising in helping unexperienced endoscopists differentiate inflammatory intestinal diseases. CONFERENCE: The abstract of this article has won the first prize of the Young Investigator Award during the Asian Pacific Digestive Week (APDW) 2019 held in Kolkata, India.

[Coagulation function and hemorrhagic diseases in preterm infants with different gestational ages].

OBJECTIVE: To study the correlation between coagulation function and gestational age in preterm infants and the possible value of coagulation function measurement in predicting hemorrhagic diseases. METHODS: The clinical data of preterm infants who were hospitalized between September 2016 and August 2017 were collected. The coagulation indicators were measured within 2 hours after birth. According to the gestational age, the preterm infants were divided into late preterm infant group (n=322), early preterm infant group (n=241) and extremely/very early preterm infant group (n=128). Coagulation function was compared among the three groups, as well as between the preterm infants with and without hemorrhagic diseases within 3 days after birth. RESULTS: There were significant differences in thrombin time (TT), prothrombin time (PT), activated partial thromboplastin time (APTT), fibrinogen degradation product (FDP) and D-dimer (DD) among the three groups (P<0.05). APTT, PT, FDP and DD were negatively correlated with gestational age, while TT was positively correlated with gestational age (P<0.05). The preterm infants with hemorrhagic diseases had a longer APTT and a higher level of DD (P<0.05). CONCLUSIONS: Coagulation function gradually becomes mature in preterm infants with the increase in gestational age. Abnormal APTT and DD indicate that preterm infants may have a higher risk of hemorrhagic diseases.

Blood concentrations of lead, cadmium, mercury and their association with biomarkers of DNA oxidative damage in preschool children living in an e-waste recycling area.

Reactive oxygen species (ROS)-induced DNA damage occurs in heavy metal exposure, but the simultaneous effect on DNA repair is unknown. We investigated the influence of co-exposure of lead (Pb), cadmium (Cd), and mercury (Hg) on 8-hydroxydeoxyguanosine (8-OHdG) and human repair enzyme 8-oxoguanine DNA glycosylase (hOGG1) mRNA levels in exposed children to evaluate the imbalance of DNA damage and repair. Children within the age range of 3-6 years from a primitive electronic waste (e-waste) recycling town were chosen as participants to represent a heavy metal-exposed population. 8-OHdG in the children's urine was assessed for heavy metal-induced oxidative effects, and the hOGG1 mRNA level in their blood represented the DNA repair ability of the children. Among the children surveyed, 88.14% (104/118) had a blood Pb level >5 µg/dL, 22.03% (26/118) had a blood Cd level >1 µg/dL, and 62.11% (59/95) had a blood Hg level >10 µg/dL. Having an e-waste workshop near the house was a risk factor contributing to high blood Pb (r s  = 0.273, p < 0.01), while Cd and Hg exposure could have come from other contaminant sources. Preschool children of fathers who had a college or university education had significantly lower 8-OHdG levels (median 242.76 ng/g creatinine, range 154.62-407.79 ng/g creatinine) than did children of fathers who had less education (p = 0.035). However, we did not observe a significant difference in the mRNA expression levels of hOGG1 between the different variables. Compared with children having low lead exposure (quartile 1), the children with high Pb exposure (quartiles 2, 3, and 4) had significantly higher 8-OHdG levels (ß Q2 = 0.362, 95% CI 0.111-0.542; ß Q3 = 0.347, 95% CI 0.103-0.531; ß Q4 = 0.314, 95% CI 0.087-0.557). Associations between blood Hg levels and 8-OHdG were less apparent. Compared with low levels of blood Hg (quartile 1), elevated blood Hg levels (quartile 2) were associated with higher 8-OHdG levels (ß Q2 = 0.236, 95% CI 0.039-0.406). Compared with children having low lead exposure (quartile 1), the children with high Pb exposure (quartiles 2, 3, and 4) had significantly higher 8-OHdG levels.

Transplantation of Wnt3a-modified neural stem cells promotes neural regeneration and functional recovery after spinal cord injury via Wnt-Gli2 pathway.

Neural stem cell (NSCs) transplantation has great potential in the treatment of spinal cord injury (SCI). Previous studies have indicated that the Wnt pathway could regulate the expression of basic helix-loop-helix (bHLH) family factor Hes5 and Mash1 in NSCs, but not through the notch intracellular domain. This suggests that there are other signals involved in this process. The aim of this study was to investigate the role of Wnt-Gli2 pathway in the treatment of SCI by transplanting neural stem cells. NSCs were isolated from the striata of embryonic day 14 mice. Activation of the Wnt pathway was achieved using Wnt3a protein, while Gli2 was inhibited using Gli2-siRNA. Expression levels of Gli2 and bHLH factors were assessed using western blotting. NSCs proliferation was evaluated using CCK-8 assay, and neural differentiation was determined by immunofluorescence staining. Finally, the modified NSCs were transplanted into mice with SCI, and their effects were assessed using behavioral and histological tests. Our results demonstrated that Wnt3a promoted the expression of Mash1 through Gli2. Moreover, the expression of Ngn1 and Hes1 was up-regulated, while Hes5 was down-regulated. Wnt3a also promoted NSCs proliferation and neural differentiation through this signaling pathway. In vivo experiments showed that NSCs transplantation mediated by Wnt3a-Gli2 signaling increased the number of neurons and resulted in improved Basso Mouse Scale scores. In conclusion, our findings suggest that Gli2 plays a role in mediating the regulation of Wnt3a signaling on promoting NSCs proliferation and neural differentiation. This pathway is therefore important in NSCs-mediated SCI recovery.

A joint learning method for incomplete and imbalanced data in electronic health record based on generative adversarial networks.

Electronic health records (EHR), present challenges of incomplete and imbalanced data in clinical predictions. Previous studies addressed these two issues with two-step separately, which caused the decrease in the performance of prediction tasks. In this paper, we propose a unified framework to simultaneously addresses the challenges of incomplete and imbalanced data in EHR. Based on the framework, we develop a model called Missing Value Imputation and Imbalanced Learning Generative Adversarial Network (MVIIL-GAN). We use MVIIL-GAN to perform joint learning on the imputation process of high missing rate data and the conditional generation process of EHR data. The joint learning is achieved by introducing two discriminators to distinguish the fake data from the generated data at sample-level and variable-level. MVIIL-GAN integrate the missing values imputation and data generation in one step, improving the consistency of parameter optimization and the performance of prediction tasks. We evaluate our framework using the public dataset MIMIC-IV with high missing rates data and imbalanced data. Experimental results show that MVIIL-GAN outperforms existing methods in prediction performance. The implementation of MVIIL-GAN can be found at https://github.com/Peroxidess/MVIIL-GAN.

3D ARCNN: An Asymmetric Residual CNN for False Positive Reduction in Pulmonary Nodule.

Lung cancer is with the highest morbidity and mortality, and detecting cancerous lesions early is essential for reducing mortality rates. Deep learning-based lung nodule detection techniques have shown better scalability than traditional methods. However, pulmonary nodule test results often include a number of false positive outcomes. In this paper, we present a novel asymmetric residual network called 3D ARCNN that leverages 3D features and spatial information of lung nodules to improve classification performance. The proposed framework uses an internally cascaded multi-level residual model for fine-grained learning of lung nodule features and multi-layer asymmetric convolution to address the problem of large neural network parameters and poor reproducibility. We evaluate the proposed framework on the LUNA16 dataset and achieve a high detection sensitivity of 91.6%, 92.7%, 93.2%, and 95.8% for 1, 2, 4, and 8 false positives per scan, respectively, with an average CPM index of 0.912. Quantitative and qualitative evaluations demonstrate the superior performance of our framework compared to existing methods. 3D ARCNN framework can effectively reduce the possibility of false positive lung nodules in the clinical.

PRTA:Joint extraction of medical nested entities and overlapping relation via parameter sharing progressive recognition and targeted assignment decoding scheme.

Nested entities and relationship extraction are two tasks for analysis of electronic medical records. However, most of existing medical information extraction models consider these tasks separately, resulting in a lack of consistency between them. In this paper, we propose a joint medical entity-relation extraction model with progressive recognition and targeted assignment (PRTA). Entities and relations share the information of sequence and word embedding layers in the joint decoding stage. They are trained simultaneously and realize information interaction by updating the shared parameters. Specifically, we design a compound triangle strategy for the nested entity recognition and an adaptive multi-space interactive strategy for relationship extraction. Then, we construct a parameter-shared information space based on semantic continuity to decode entities and relationships. Extensive experiments were conducted on the Private Liver Disease Dataset (PLDD) provided by Beijing Friendship Hospital of Capital Medical University and public datasets (NYT, ACE04 and ACE05). The results show that our method outperforms existing SOTA methods in most indicators, and effectively handles nested entities and overlapping relationships.

Cardio-respiratory motion compensation for coronary roadmapping in fluoroscopic imaging.

BACKGROUND: Inferring the shape and position of coronary artery poses challenges when using fluoroscopic image guidance during percutaneous coronary intervention (PCI) procedure. Although angiography enables coronary artery visualization, the use of injected contrast agent raises concerns about radiation exposure and the risk of contrast-induced nephropathy. To address these issues, dynamic coronary roadmapping overlaid on fluoroscopic images can provide coronary visual feedback without contrast injection. PURPOSE: This paper proposes a novel cardio-respiratory motion compensation method that utilizes cardiac state synchronization and catheter motion estimation to achieve coronary roadmapping in fluoroscopic images. METHODS: For more accurate cardiac state synchronization, video frame interpolation is applied to increase the frame rate of the original limited angiographic images, resulting in higher framerate and more adequate roadmaps. The proposed method also incorporates a multi-length cross-correlation based adaptive electrocardiogram (ECG) matching to address irregular cardiac motion situation. Furthermore, a shape-constrained path searching method is proposed to extract catheter structure from both fluoroscopic and angiographic image. Then catheter motion is estimated using a cascaded matching approach with an outlier removal strategy, leading to a final corrected roadmap. RESULTS: Evaluation of the proposed method on clinical x-ray images demonstrates its effectiveness, achieving a 92.8% F1 score for catheter extraction on 589 fluoroscopic and angiographic images. Additionally, the method achieves a 5.6-pixel distance error of the coronary roadmap on 164 intraoperative fluoroscopic images. CONCLUSIONS: Overall, the proposed method achieves accurate coronary roadmapping in fluoroscopic images and shows potential to overlay accurate coronary roadmap on fluoroscopic image in assisting PCI.

Embedding-Alignment Fusion-Based Graph Convolution Network With Mixed Learning Strategy for 4D Medical Image Reconstruction.

In recent years, 4D medical image involving structural and motion information of tissue has attracted increasing attention. The key to the 4D image reconstruction is to stack the 2D slices based on matching the aligned motion states. In this study, the distribution of the 2D slices with the different motion states is modeled as a manifold graph, and the reconstruction is turned to be the graph alignment. An embedding-alignment fusion-based graph convolution network (GCN) with a mixed-learning strategy is proposed to align the graphs. Herein, the embedding and alignment processes of graphs interact with each other to realize a precise alignment with retaining the manifold distribution. The mixed strategy of self- and semi-supervised learning makes the alignment sparse to avoid the mismatching caused by outliers in the graph. In the experiment, the proposed 4D reconstruction approach is validated on the different modalities including Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and Ultrasound (US). We evaluate the reconstruction accuracy and compare it with those of state-of-the-art methods. The experiment results demonstrate that our approach can reconstruct a more accurate 4D image.

Robot-assisted augmented reality surgical navigation based on optical tracking for mandibular reconstruction surgery.

PURPOSE: This work proposes a robot-assisted augmented reality (AR) surgical navigation system for mandibular reconstruction. The system accurately superimposes the preoperative osteotomy plan of the mandible and fibula into a real scene. It assists the doctor in osteotomy quickly and safely under the guidance of the robotic arm. METHODS: The proposed system mainly consists of two modules: the AR guidance module of the mandible and fibula and the robot navigation module. In the AR guidance module, we propose an AR calibration method based on the spatial registration of the image tracking marker to superimpose the virtual models of the mandible and fibula into the real scene. In the robot navigation module, the posture of the robotic arm is first calibrated under the tracking of the optical tracking system. The robotic arm can then be positioned at the planned osteotomy after the registration of the computed tomography image and the patient position. The combined guidance of AR and robotic arm can enhance the safety and precision of the surgery. RESULTS: The effectiveness of the proposed system was quantitatively assessed on cadavers. In the AR guidance module, osteotomies of the mandible and fibula achieved mean errors of 1.61 ± 0.62 and 1.08 ± 0.28 mm, respectively. The mean reconstruction error of the mandible was 1.36 ± 0.22 mm. In the AR-robot guidance module, the mean osteotomy errors of the mandible and fibula were 1.47 ± 0.46 and 0.98 ± 0.24 mm, respectively. The mean reconstruction error of the mandible was 1.20 ± 0.36 mm. CONCLUSIONS: The cadaveric experiments of 12 fibulas and six mandibles demonstrate the proposed system's effectiveness and potential clinical value in reconstructing the mandibular defect with a free fibular flap.

Facial augmented reality based on hierarchical optimization of similarity aspect graph.

BACKGROUND: The existing face matching method requires a point cloud to be drawn on the real face for registration, which results in low registration accuracy due to the irregular deformation of the patient's skin that makes the point cloud have many outlier points. METHODS: This work proposes a non-contact pose estimation method based on similarity aspect graph hierarchical optimization. The proposed method constructs a distance-weighted and triangular-constrained similarity measure to describe the similarity between views by automatically identifying the 2D and 3D feature points of the face. A mutual similarity clustering method is proposed to construct a hierarchical aspect graph with 3D pose as nodes. A Monte Carlo tree search strategy is used to search the hierarchical aspect graph for determining the optimal pose of the facial 3D model, so as to realize the accurate registration of the facial 3D model and the real face. RESULTS: The proposed method was used to conduct accuracy verification experiments on the phantoms and volunteers, which were compared with four advanced pose calibration methods. The proposed method obtained average fusion errors of 1.13 ± 0.20 mm and 0.92 ± 0.08 mm in head phantom and volunteer experiments, respectively, which exhibits the best fusion performance among all comparison methods. CONCLUSIONS: Our experiments proved the effectiveness of the proposed pose estimation method in facial augmented reality.