Your browser doesn't support javascript.
loading
Assessing the Performance of Artificial Intelligence Models: Insights from the American Society of Functional Neuroradiology Artificial Intelligence Competition.
Jiang, Bin; Ozkara, Burak Berksu; Zhu, Guangming; Boothroyd, Derek; Allen, Jason W; Barboriak, Daniel P; Chang, Peter; Chan, Cynthia; Chaudhari, Ruchir; Chen, Hui; Chukus, Anjeza; Ding, Victoria; Douglas, David; Filippi, Christopher G; Flanders, Adam E; Godwin, Ryan; Hashmi, Syed; Hess, Christopher; Hsu, Kevin; Lui, Yvonne W; Maldjian, Joseph A; Michel, Patrik; Nalawade, Sahil S; Patel, Vishal; Raghavan, Prashant; Sair, Haris I; Tanabe, Jody; Welker, Kirk; Whitlow, Christopher T; Zaharchuk, Greg; Wintermark, Max.
Afiliação
  • Jiang B; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Ozkara BB; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Zhu G; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Boothroyd D; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Allen JW; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Barboriak DP; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Chang P; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Chan C; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Chaudhari R; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Chen H; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Chukus A; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Ding V; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Douglas D; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Filippi CG; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Flanders AE; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Godwin R; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Hashmi S; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Hess C; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Hsu K; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Lui YW; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Maldjian JA; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Michel P; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Nalawade SS; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Patel V; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Raghavan P; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Sair HI; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Tanabe J; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Welker K; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Whitlow CT; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Zaharchuk G; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
  • Wintermark M; From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Z.), and Department of Medicine (D.B., V.D.), Stanford University, Stanford, CA, USA; Department of Neuroradiology (B.B.O., H.C., M.W.), MD Anderson Cancer Center, Houston, TX, USA; Department of Neurology (G.Z.), University of
AJNR Am J Neuroradiol ; 2024 May 03.
Article em En | MEDLINE | ID: mdl-38663992
ABSTRACT
BACKGROUND AND

PURPOSE:

Artificial intelligence (AI) models in radiology are frequently developed and validated using datasets from a single institution and are rarely tested on independent, external datasets, raising questions about their generalizability and applicability in clinical practice. The American Society of Functional Neuroradiology (ASFNR) organized a multi-center AI competition to evaluate the proficiency of developed models in identifying various pathologies on NCCT, assessing age-based normality and estimating medical urgency. MATERIALS AND

METHODS:

In total, 1201 anonymized, full-head NCCT clinical scans from five institutions were pooled to form the dataset. The dataset encompassed normal studies as well as pathologies including acute ischemic stroke, intracranial hemorrhage, traumatic brain injury, and mass effect (detection of these-task 1). NCCTs were also assessed to determine if findings were consistent with expected brain changes for the patient's age (task 2 age-based normality assessment) and to identify any abnormalities requiring immediate medical attention (task 3 evaluation of findings for urgent intervention). Five neuroradiologists labeled each NCCT, with consensus interpretations serving as the ground truth. The competition was announced online, inviting academic institutions and companies. Independent central analysis assessed each model's performance. Accuracy, sensitivity, specificity, positive and negative predictive values, and receiver operating characteristic (ROC) curves were generated for each AI model, along with the area under the ROC curve (AUROC).

RESULTS:

1177 studies were processed by four teams. The median age of patients was 62, with an interquartile range of 33. 19 teams from various academic institutions registered for the competition. Of these, four teams submitted their final results. No commercial entities participated in the competition. For task 1, AUROCs ranged from 0.49 to 0.59. For task 2, two teams completed the task with AUROC values of 0.57 and 0.52. For task 3, teams had little to no agreement with the ground truth.

CONCLUSIONS:

To assess the performance of AI models in real-world clinical scenarios, we analyzed their performance in the ASFNR AI Competition. The first ASFNR Competition underscored the gap between expectation and reality; the models largely fell short in their assessments. As the integration of AI tools into clinical workflows increases, neuroradiologists must carefully recognize the capabilities, constraints, and consistency of these technologies. Before institutions adopt these algorithms, thorough validation is essential to ensure acceptable levels of performance in clinical settings.ABBREVIATIONS AI = artificial intelligence; ASFNR = American Society of Functional Neuroradiology; AUROC = area under the receiver operating characteristic curve; DICOM = Digital Imaging and Communications in Medicine; GEE = generalized estimation equation; IQR = interquartile range; NPV = negative predictive value; PPV = positive predictive value; ROC = receiver operating characteristic; TBI = traumatic brain injury.
Texto completo
Imprimir
XML
PubMed Links
Buscar no Google

Texto completo: 1 Coleções: 01-internacional Base de dados: MEDLINE Idioma: En Revista: AJNR Am J Neuroradiol Ano de publicação: 2024 Tipo de documento: Article
Texto completo
Imprimir
XML
PubMed Links
Buscar no Google

Texto completo: 1 Coleções: 01-internacional Base de dados: MEDLINE Idioma: En Revista: AJNR Am J Neuroradiol Ano de publicação: 2024 Tipo de documento: Article