Performance and Agreement When Annotating Chest X-ray Text Reports-A Preliminary Step in the Development of a Deep Learning-Based Prioritization and Detection System.

A chest X-ray report is a communicative tool and can be used as data for developing artificial intelligence-based decision support systems. For both, consistent understanding and labeling is important. Our aim was to investigate how readers would comprehend and annotate 200 chest X-ray reports. Reports written between 1 January 2015 and 11 March 2022 were selected based on search words. Annotators included three board-certified radiologists, two trained radiologists (physicians), two radiographers (radiological technicians), a non-radiological physician, and a medical student. Consensus labels by two or more of the experienced radiologists were considered "gold standard". Matthew's correlation coefficient (MCC) was calculated to assess annotation performance, and descriptive statistics were used to assess agreement between individual annotators and labels. The intermediate radiologist had the best correlation to "gold standard" (MCC 0.77). This was followed by the novice radiologist and medical student (MCC 0.71 for both), the novice radiographer (MCC 0.65), non-radiological physician (MCC 0.64), and experienced radiographer (MCC 0.57). Our findings showed that for developing an artificial intelligence-based support system, if trained radiologists are not available, annotations from non-radiological annotators with basic and general knowledge may be more aligned with radiologists compared to annotations from sub-specialized medical staff, if their sub-specialization is outside of diagnostic radiology.

Inter- and Intra-Observer Agreement When Using a Diagnostic Labeling Scheme for Annotating Findings on Chest X-rays-An Early Step in the Development of a Deep Learning-Based Decision Support System.

Consistent annotation of data is a prerequisite for the successful training and testing of artificial intelligence-based decision support systems in radiology. This can be obtained by standardizing terminology when annotating diagnostic images. The purpose of this study was to evaluate the annotation consistency among radiologists when using a novel diagnostic labeling scheme for chest X-rays. Six radiologists with experience ranging from one to sixteen years, annotated a set of 100 fully anonymized chest X-rays. The blinded radiologists annotated on two separate occasions. Statistical analyses were done using Randolph's kappa and PABAK, and the proportions of specific agreements were calculated. Fair-to-excellent agreement was found for all labels among the annotators (Randolph's Kappa, 0.40-0.99). The PABAK ranged from 0.12 to 1 for the two-reader inter-rater agreement and 0.26 to 1 for the intra-rater agreement. Descriptive and broad labels achieved the highest proportion of positive agreement in both the inter- and intra-reader analyses. Annotating findings with specific, interpretive labels were found to be difficult for less experienced radiologists. Annotating images with descriptive labels may increase agreement between radiologists with different experience levels compared to annotation with interpretive labels.

Simulation-Based Training of Ultrasound-Guided Procedures in Radiology - A Systematic Review.

PURPOSE: To perform a systematic review of the effect of simulation-based training (SBT) of percutaneous abdominal and thoracic ultrasound-guided procedures and to assess the transfer of procedural competence to a clinical context. MATERIALS AND METHODS: This systematic review was conducted in accordance with the PRISMA statement. Pubmed, Embase, Web of Science, and the Cochrane Library were searched for studies assessing procedural competence after SBT. Two authors independently reviewed all studies and extracted data. Risk of bias was assessed using the Cochrane tool for randomized studies (RoB) and non-randomized studies (ROBINS-I). Quality of evidence was assessed using the GRADE approach. RESULTS: 42 studies were included. 6 were randomized controlled, 3 non-randomized controlled, and 33 non-randomized non-controlled. 26 studies examined US-guided abdominal procedures, 13 examined thoracic procedures, and 3 examined both. The results favored SBT compared to other educational interventions and found that training was superior to no training. Only two studies examined the transfer of procedural skills to a clinical context. All studies had a high or critical risk of bias. Thus, the quality of evidence for the effect of SBT on procedural competence was low, and evidence for its transfer to a clinical context was very low. CONCLUSION: The evidence supporting SBT of percutaneous abdominal and thoracic US-guided procedures remains insufficient due to methodological problems and a high risk of bias. Future studies should be randomized and single-blinded, use assessment tools supported by validity evidence, compare different educational strategies, and examine the transfer of skills to a clinical setting.