Visual Recalibration and Gating Enhancement Network for Radiology Report Generation.

Automatic radiology medical report generation is a necessary development of artificial intelligence technology in the health care. This technology serves to aid doctors in producing comprehensive diagnostic reports, alleviating the burdensome workloads of medical professionals. However, there are some challenges in generating radiological reports: (1) visual and textual data biases and (2) long-distance dependency problem. To tackle these issues, we design a visual recalibration and gating enhancement network (VRGE), which composes of the visual recalibration module and the gating enhancement module (gating enhancement module, GEM). Specifically, the visual recalibration module enhances the recognition of abnormal features in lesion areas of medical images. The GEM dynamically adjusts the contextual information in the report by introducing gating mechanisms, focusing on capturing professional medical terminology in medical text reports. We have conducted sufficient experiments on the public datasets of IU X-Ray to illustrate that the VRGE outperforms existing models.

MKCL: Medical Knowledge with Contrastive Learning model for radiology report generation.

Automatic radiology report generation has the potential to alert inexperienced radiologists to misdiagnoses or missed diagnoses and improve healthcare delivery efficiency by reducing the documentation workload of radiologists. Motivated by the continuous development of automatic image captioning, more and more deep learning methods have been proposed for automatic radiology report generation. However, the visual and textual data bias problem still face many challenges in the medical domain. Additionally, do not integrate medical knowledge, ignoring the mutual influences between medical findings, and abundant unlabeled medical images influence the accuracy of generating report. In this paper, we propose a Medical Knowledge with Contrastive Learning model (MKCL) to enhance radiology report generation. The proposed model MKCL uses IU Medical Knowledge Graph (IU-MKG) to mine the relationship among medical findings and improve the accuracy of identifying positive diseases findings from radiologic medical images. In particular, we design Knowledge Enhanced Attention (KEA), which integrates the IU-MKG and the extracted chest radiological visual features to alleviate textual data bias. Meanwhile, this paper leverages supervised contrastive learning to relieve radiographic medical images which have not been labeled, and identify abnormalities from images. Experimental results on the public dataset IU X-ray show that our proposed model MKCL outperforms other state-of-the-art report generation methods. Ablation studies also demonstrate that IU medical knowledge graph module and supervised contrastive learning module enhance the ability of the model to detect the abnormal parts and accurately describe the abnormal findings. The source code is available at: https://github.com/Eleanorhxd/MKCL.