Performance of Multiple Pretrained BERT Models to Automate and Accelerate Data Annotation for Large Datasets.

Purpose: To develop and evaluate domain-specific and pretrained bidirectional encoder representations from transformers (BERT) models in a transfer learning task on varying training dataset sizes to annotate a larger overall dataset. Materials and Methods: The authors retrospectively reviewed 69 095 anonymized adult chest radiograph reports (reports dated April 2020-March 2021). From the overall cohort, 1004 reports were randomly selected and labeled for the presence or absence of each of the following devices: endotracheal tube (ETT), enterogastric tube (NGT, or Dobhoff tube), central venous catheter (CVC), and Swan-Ganz catheter (SGC). Pretrained transformer models (BERT, PubMedBERT, DistilBERT, RoBERTa, and DeBERTa) were trained, validated, and tested on 60%, 20%, and 20%, respectively, of these reports through fivefold cross-validation. Additional training involved varying dataset sizes with 5%, 10%, 15%, 20%, and 40% of the 1004 reports. The best-performing epochs were used to assess area under the receiver operating characteristic curve (AUC) and determine run time on the overall dataset. Results: The highest average AUCs from fivefold cross-validation were 0.996 for ETT (RoBERTa), 0.994 for NGT (RoBERTa), 0.991 for CVC (PubMedBERT), and 0.98 for SGC (PubMedBERT). DeBERTa demonstrated the highest AUC for each support device trained on 5% of the training set. PubMedBERT showed a higher AUC with a decreasing training set size compared with BERT. Training and validation time was shortest for DistilBERT at 3 minutes 39 seconds on the annotated cohort. Conclusion: Pretrained and domain-specific transformer models required small training datasets and short training times to create a highly accurate final model that expedites autonomous annotation of large datasets.Keywords: Informatics, Named Entity Recognition, Transfer Learning Supplemental material is available for this article. ©RSNA, 2022See also the commentary by Zech in this issue.

Facilitating Surveillance of Incidental Findings Using a Novel Reporting Template: Proof of Concept in Patients With Pancreatic Abnormalities.

OBJECTIVE: To determine the surveillance impact of utilizing a discrete field in structured radiology reports in patients with incidental pancreatic findings. METHODS: We implemented a dictation template containing a discrete structured field element to auto-trigger listing of patients with incidental pancreatic findings on a pancreas clinic registry in the electronic health record. We isolated CT and MRI reports with incidental pancreatic findings over a 24-month period. We stratified patients by presence or absence of the discrete field element in reports (flagged versus unflagged) and evaluated the impact of report flagging on likelihood of clinic follow-up, follow-up imaging, endoscopic ultrasound, surgical intervention, genetics referral, obtaining pathologic diagnosis, and time interval between index imaging to various outcomes. RESULTS: Patients with flagged reports were more likely to be seen or discussed in a pancreas clinic compared with those with unflagged reports (189 of 376, 50.3% versus 79 of 474, 16.7%; P <. 001). Patients with flagged reports were more likely to get follow-up imaging than patients with unflagged reports (188 of 376, 50.0% versus 121 of 474, 25.5%; P < .001) and were more likely to undergo appropriate management of actionable findings compared with patients in the unflagged group (23 of 62, 37.1% versus 28 of 129, 21.7%; P = .036). DISCUSSION: Implementation of a structured discrete field element for reporting of patients with incidental pancreatic findings had positive impact on surveillance measures and can be applied in other organ systems with established surveillance guidelines to standardize patient care.