Development and evaluation of a program based on a generative pre-trained transformer model from a public natural language processing platform for efficiency enhancement in post-procedural quality control of esophageal endoscopic submucosal dissection.

ABSTRACT

BACKGROUND: Post-procedural quality control of endoscopic submucosal dissection (ESD) is emphasized in guidelines. However, this process can be tedious and time-consuming. Recently, a pre-training model called generative pre-trained transformer (GPT) on a public natural language processing platform has emerged and garnered significant attention, whose capabilities align well with the post-procedural quality control process and have the potential to streamline it. Therefore, we developed a simple program utilizing this platform and evaluated its performance. METHODS: Esophageal ESDs were retrospectively included. The manual quality control process was performed and act as reference standard. GPT's prompt was optimized through multiple iterations. A Python program was developed to automatically submit prompt with pathological report of each ESD procedure and collect quality control information provided by GPT. Its performance on quality control was evaluated with accuracy, precision, recall, and F-1 score. RESULTS: 165 cases were involved into the dataset, of which 5 were utilized as the prompt optimization dataset and 160 as the validation dataset. Definitive prompt was achieved through seven iterations. Time spent on the validation dataset by GPT was 13.47 ± 2.43 min. Accuracies of pathological diagnosis, invasion depth, horizontal margin, vertical margin, vascular invasion, and lymphatic invasion of the quality control program were (0.940, 0.952) (95% CI), (0.925, 0.945) (95% CI), 0.931, 1.0, and 1.0, respectively. Precisions were (0.965, 0.969) (95% CI), (0.934, 0.954) (95% CI), and 0.957 for pathological diagnosis, invasion depth, and horizontal margin, respectively. Recalls were (0.940, 0.952) (95% CI), (0.925, 0.945) (95% CI), and 0.931 for factors as mentioned, respectively. F1-score were (0.945, 0.957) (95% CI), (0.928, 0.948) (95% CI), and 0.941 for factors as mentioned, respectively. CONCLUSIONS: This quality control program was qualified of post-procedural quality control of esophageal ESDs. GPT can be easily applied to this quality control process and reduce workload of the endoscopists.