ABSTRACT
Objectives: The aim of this study was to evaluate the diagnostic performance of nonspecialist readers with and without the use of an artificial intelligence (AI) support tool to detect traumatic fractures on radiographs of the appendicular skeleton. Methods: The design was a retrospective, fully crossed multi-reader, multi-case study on a balanced dataset of patients (≥2 years of age) with an AI tool as a diagnostic intervention. Fifteen readers assessed 340 radiographic exams, with and without the AI tool in 2 different sessions and the time spent was automatically recorded. Reference standard was established by 3 consultant radiologists. Sensitivity, specificity, and false positives per patient were calculated. Results: Patient-wise sensitivity increased from 72% to 80% (P < .05) and patient-wise specificity increased from 81% to 85% (P < .05) in exams aided by the AI tool compared to the unaided exams. The increase in sensitivity resulted in a relative reduction of missed fractures of 29%. The average rate of false positives per patient decreased from 0.16 to 0.14, corresponding to a relative reduction of 21%. There was no significant difference in average reading time spent per exam. The largest gain in fracture detection performance, with AI support, across all readers, was on nonobvious fractures with a significant increase in sensitivity of 11 percentage points (pp) (60%-71%). Conclusions: The diagnostic performance for detection of traumatic fractures on radiographs of the appendicular skeleton improved among nonspecialist readers tested AI fracture detection support tool showed an overall reader improvement in sensitivity and specificity when supported by an AI tool. Improvement was seen in both sensitivity and specificity without negatively affecting the interpretation time. Advances in knowledge: The division and analysis of obvious and nonobvious fractures are novel in AI reader comparison studies like this.
ABSTRACT
PURPOSE: To externally validate an artificial intelligence (AI) tool for radiographic knee osteoarthritis severity classification on a clinical dataset. METHOD: This retrospective, consecutive patient sample, external validation study used weight-bearing, non-fixed-flexion posterior-anterior knee radiographs from a clinical production PACS. The index test was ordinal Kellgren-Lawrence grading by an AI tool, two musculoskeletal radiology consultants, two reporting technologists, and two resident radiologists. Grading was repeated by all readers after at least four weeks. Reference test was the consensus of the two consultants. The primary outcome was quadratic weighted kappa. Secondary outcomes were ordinal weighted accuracy, multiclass accuracy and F1-score. RESULTS: 50 consecutive patients between September 24, 2019 and October 22, 2019 were retrospectively included (3 excluded) totaling 99 knees (1 excluded). Quadratic weighted kappa for the AI tool and the consultant consensus was 0.88 CI95% (0.82-0.92). Agreement between the consultants was 0.89 CI95% (0.85-0.93). Intra-rater agreements for the consultants were 0.96 CI95% (0.94-0.98) and 0.94 CI95% (0.91-0.96) respectively. For the AI tool it was 1 CI95% (1-1). For the AI tool, ordinal weighted accuracy was 97.8% CI95% (96.9-98.6 %). Average multiclass accuracy and F1-score were 84% (83/99) CI95% (77-91%) and 0.67 CI95% (0.51-0.81). CONCLUSIONS: The AI tool achieved the same good-to-excellent agreement with the radiology consultant consensus for radiographic knee osteoarthritis severity classification as the consultants did with each other.