Your browser doesn't support javascript.
loading
A Multimodal Transformer Model for Recognition of Images from Complex Laparoscopic Surgical Videos.
Abiyev, Rahib H; Altabel, Mohamad Ziad; Darwish, Manal; Helwan, Abdulkader.
Afiliação
  • Abiyev RH; Applied Artificial Intelligence Research Centre, Department of Computer Engineering, Near East University, 99132 North Cyprus, Turkey.
  • Altabel MZ; Applied Artificial Intelligence Research Centre, Department of Computer Engineering, Near East University, 99132 North Cyprus, Turkey.
  • Darwish M; Applied Artificial Intelligence Research Centre, Department of Computer Engineering, Near East University, 99132 North Cyprus, Turkey.
  • Helwan A; Department of Health, Medicine and Caring Sciences, Linköping University, 581 85 Linköping, Sweden.
Diagnostics (Basel) ; 14(7)2024 Mar 23.
Article em En | MEDLINE | ID: mdl-38611594
ABSTRACT
The determination of the potential role and advantages of artificial intelligence-based models in the field of surgery remains uncertain. This research marks an initial stride towards creating a multimodal model, inspired by the Video-Audio-Text Transformer, that aims to reduce negative occurrences and enhance patient safety. The model employs text and image embedding state-of-the-art models (ViT and BERT) to assess their efficacy in extracting the hidden and distinct features from the surgery video frames. These features are then used as inputs for convolution-free Transformer architectures to extract comprehensive multidimensional representations. A joint space is then used to combine the text and image features extracted from both Transformer encoders. This joint space ensures that the relationships between the different modalities are preserved during the combination process. The entire model was trained and tested on laparoscopic cholecystectomy (LC) videos encompassing various levels of complexity. Experimentally, a mean accuracy of 91.0%, a precision of 81%, and a recall of 83% were reached by the model when tested on 30 videos out of 80 from the Cholec80 dataset.
Palavras-chave

Texto completo: 1 Base de dados: MEDLINE Idioma: En Ano de publicação: 2024 Tipo de documento: Article

Texto completo: 1 Base de dados: MEDLINE Idioma: En Ano de publicação: 2024 Tipo de documento: Article