External validation of the RSNA 2020 pulmonary embolism detection challenge winning deep learning algorithm.

PURPOSE: To evaluate the diagnostic performance and generalizability of the winning DL algorithm of the RSNA 2020 PE detection challenge to a local population using CTPA data from two hospitals. MATERIALS AND METHODS: Consecutive CTPA images from patients referred for suspected PE were retrospectively analysed. The winning RSNA 2020 DL algorithm was retrained on the RSNA-STR Pulmonary Embolism CT (RSPECT) dataset. The algorithm was tested in hospital A on multidetector CT (MDCT) images of 238 patients and in hospital B on spectral detector CT (SDCT) and virtual monochromatic images (VMI) of 114 patients. The output of the DL algorithm was compared with a reference standard, which included a consensus reading by at least two experienced cardiothoracic radiologists for both hospitals. Areas under the receiver operating characteristic curve (AUCs) were calculated. Sensitivity and specificity were determined using the maximum Youden index. RESULTS: According to the reference standard, PE was present in 73 patients (30.7%) in hospital A and 33 patients (29.0%) in hospital B. For the DL algorithm the AUC was 0.96 (95% CI 0.92-0.98) in hospital A, 0.89 (95% CI 0.81-0.94) for conventional reconstruction in hospital B and 0.87 (95% CI 0.80-0.93) for VMI. CONCLUSION: The RSNA 2020 pulmonary embolism detection on CTPA challenge winning DL algorithm, retrained on the RSPECT dataset, showed high diagnostic accuracy on MDCT images. A somewhat lower performance was observed on SDCT images, which suggest additional training on novel CT technology may improve generalizability of this DL algorithm.

Diagnostic accuracy of an artificial intelligence algorithm versus radiologists for fracture detection on cervical spine CT.

OBJECTIVES: To compare diagnostic accuracy of a deep learning artificial intelligence (AI) for cervical spine (C-spine) fracture detection on CT to attending radiologists and assess which undetected fractures were injuries in need of stabilising therapy (IST). METHODS: This single-centre, retrospective diagnostic accuracy study included consecutive patients (age ≥18 years; 2007-2014) screened for C-spine fractures with CT. To validate ground truth, one radiologist and three neurosurgeons independently examined scans positive for fracture. Negative scans were followed up until 2022 through patient files and two radiologists reviewed negative scans that were flagged positive by AI. The neurosurgeons determined which fractures were ISTs. Diagnostic accuracy of AI and attending radiologists (index tests) were compared using McNemar. RESULTS: Of the 2368 scans (median age, 48, interquartile range 30-65; 1441 men) analysed, 221 (9.3%) scans contained C-spine fractures with 133 IST. AI detected 158/221 scans with fractures (sensitivity 71.5%, 95% CI 65.5-77.4%) and 2118/2147 scans without fractures (specificity 98.6%, 95% CI 98.2-99.1). In comparison, attending radiologists detected 195/221 scans with fractures (sensitivity 88.2%, 95% CI 84.0-92.5%, p < 0.001) and 2130/2147 scans without fracture (specificity 99.2%, 95% CI 98.8-99.6, p = 0.07). Of the fractures undetected by AI 30/63 were ISTs versus 4/26 for radiologists. AI detected 22/26 fractures undetected by the radiologists, including 3/4 undetected ISTs. CONCLUSION: Compared to attending radiologists, the artificial intelligence has a lower sensitivity and a higher miss rate of fractures in need of stabilising therapy; however, it detected most fractures undetected by the radiologists, including fractures in need of stabilising therapy. Clinical relevance statement The artificial intelligence algorithm missed more cervical spine fractures on CT than attending radiologists, but detected 84.6% of fractures undetected by radiologists, including fractures in need of stabilising therapy. KEY POINTS: The impact of artificial intelligence for cervical spine fracture detection on CT on fracture management is unknown. The algorithm detected less fractures than attending radiologists, but detected most fractures undetected by the radiologists including almost all in need of stabilising therapy. The artificial intelligence algorithm shows potential as a concurrent reader.

Added value of an artificial intelligence algorithm in reducing the number of missed incidental acute pulmonary embolism in routine portal venous phase chest CT.

OBJECTIVES: The purpose of this study was to evaluate the incremental value of artificial intelligence (AI) compared to the diagnostic accuracy of radiologists alone in detecting incidental acute pulmonary embolism (PE) on routine portal venous contrast-enhanced chest computed tomography (CT). METHODS: CTs of 3089 consecutive patients referred to the radiology department for a routine contrast-enhanced chest CT between 27-5-2020 and 31-12-2020, were retrospectively analysed by a CE-certified and FDA-approved AI algorithm. The diagnostic performance of the AI was compared to the initial report. To determine the reference standard, discordant findings were independently evaluated by two readers. In case of disagreement, another experienced cardiothoracic radiologist with knowledge of the initial report and the AI output adjudicated. RESULTS: The prevalence of acute incidental PE in the reference standard was 2.2% (67 of 3089 patients). In 25 cases, AI detected initially unreported PE. This included three cases concerning central/lobar PE. Sensitivity of the AI algorithm was significantly higher than the outcome of the initial report (respectively 95.5% vs. 62.7%, p < 0.001), whereas specificity was very high for both (respectively 99.6% vs 99.9%, p = 0.012). The AI algorithm only showed a slightly higher amount of false-positive findings (11 vs. 2), resulting in a significantly lower PPV (85.3% vs. 95.5%, p = 0.047). CONCLUSION: The AI algorithm showed high diagnostic accuracy in diagnosing incidental PE, detecting an additional 25 cases of initially unreported PE, accounting for 37.3% of all positive cases. CLINICAL RELEVANCE STATEMENT: Radiologist support from AI algorithms in daily practice can prevent missed incidental acute PE on routine chest CT, without a high burden of false-positive cases. KEY POINTS: • Incidental pulmonary embolism is often missed by radiologists in non-diagnostic scans with suboptimal contrast opacification within the pulmonary trunk. • An artificial intelligence algorithm showed higher sensitivity detecting incidental pulmonary embolism on routine portal venous chest CT compared to the initial report. • Implementation of artificial intelligence support in routine daily practice will reduce the number of missed incidental pulmonary embolism.

An artificial intelligence algorithm for pulmonary embolism detection on polychromatic computed tomography: performance on virtual monochromatic images.

OBJECTIVES: Virtual monochromatic images (VMI) are increasingly used in clinical practice as they improve contrast-to-noise ratio. However, due to their different appearances, the performance of artificial intelligence (AI) trained on conventional CT images may worsen. The goal of this study was to assess the performance of an established AI algorithm trained on conventional polychromatic computed tomography (CT) images (CPI) to detect pulmonary embolism (PE) on VMI. METHODS: Paired 60 kiloelectron volt (keV) VMI and CPI of 114 consecutive patients suspected of PE, obtained with a detector-based spectral CT scanner, were retrospectively analyzed by an established AI algorithm. The CT pulmonary angiography (CTPA) were classified as positive or negative for PE on a per-patient level. The reference standard was established using a comprehensive method that combined the evaluation of the attending radiologist and three experienced cardiothoracic radiologists aided by two different detection tools. Sensitivity, specificity, positive and negative predictive values and likelihood ratios of the algorithm on VMI and CPI were compared. RESULTS: The prevalence of PE according to the reference standard was 35.1% (40 patients). None of the diagnostic accuracy measures of the algorithm showed a significant difference between CPI and VMI. Sensitivity was 77.5% (95% confidence interval (CI) 64.6-90.4%) and 85.0% (73.9-96.1%) (p = 0.08) on CPI and VMI respectively and specificity 96.0% (91.4-100.0%) and 94.6% (89.4-99.7%) (p = 0.32). CONCLUSIONS: Diagnostic performance of the AI algorithm that was trained on CPI did not drop on VMI, which is reassuring for its use in clinical practice. CLINICAL RELEVANCE STATEMENT: A commercially available AI algorithm, trained on conventional polychromatic CTPA, could be safely used on virtual monochromatic images. This supports the sustainability of AI-aided detection of PE on CT despite ongoing technological advances in medical imaging, although monitoring in daily practice will remain important. KEY POINTS: • Diagnostic accuracy of an AI algorithm trained on conventional polychromatic images to detect PE did not drop on virtual monochromatic images. • Our results are reassuring as innovations in hardware and reconstruction in CT are continuing, whilst commercial AI algorithms that are trained on older generation data enter healthcare.

Retrospective batch analysis to evaluate the diagnostic accuracy of a clinically deployed AI algorithm for the detection of acute pulmonary embolism on CTPA.

PURPOSE: To generate and extend the evidence on the clinical validity of an artificial intelligence (AI) algorithm to detect acute pulmonary embolism (PE) on CT pulmonary angiography (CTPA) of patients suspected of PE and to evaluate the possibility of reducing the risk of missed findings in clinical practice with AI-assisted reporting. METHODS: Consecutive CTPA scan data of 3316 patients referred because of suspected PE between 24-2-2018 and 31-12-2020 were retrospectively analysed by a CE-certified and FDA-approved AI algorithm. The output of the AI was compared with the attending radiologists' report. To define the reference standard, discordant findings were independently evaluated by two readers. In case of disagreement, an experienced cardiothoracic radiologist adjudicated. RESULTS: According to the reference standard, PE was present in 717 patients (21.6%). PE was missed by the AI in 23 patients, while the attending radiologist missed 60 PE. The AI detected 2 false positives and the attending radiologist 9. The sensitivity for the detection of PE by the AI algorithm was significantly higher compared to the radiology report (96.8% vs. 91.6%, p < 0.001). Specificity of the AI was also significantly higher (99.9% vs. 99.7%, p = 0.035). NPV and PPV of the AI were also significantly higher than the radiology report. CONCLUSION: The AI algorithm showed a significantly higher diagnostic accuracy for the detection of PE on CTPA compared to the report of the attending radiologist. This finding indicates that missed positive findings could be prevented with the implementation of AI-assisted reporting in daily clinical practice. CRITICAL RELEVANCE STATEMENT: Missed positive findings on CTPA of patients suspected of pulmonary embolism can be prevented with the implementation of AI-assisted care. KEY POINTS: The AI algorithm showed excellent diagnostic accuracy detecting PE on CTPA. Accuracy of the AI was significantly higher compared to the attending radiologist. Highest diagnostic accuracy can likely be achieved by radiologists supported by AI. Our results indicate that implementation of AI-assisted reporting could reduce the number of missed positive findings.