Conceptual review of outcome metrics and measures used in clinical evaluation of artificial intelligence in radiology.

Artificial intelligence (AI) has numerous applications in radiology. Clinical research studies to evaluate the AI models are also diverse. Consequently, diverse outcome metrics and measures are employed in the clinical evaluation of AI, presenting a challenge for clinical radiologists. This review aims to provide conceptually intuitive explanations of the outcome metrics and measures that are most frequently used in clinical research, specifically tailored for clinicians. While we briefly discuss performance metrics for AI models in binary classification, detection, or segmentation tasks, our primary focus is on less frequently addressed topics in published literature. These include metrics and measures for evaluating multiclass classification; those for evaluating generative AI models, such as models used in image generation or modification and large language models; and outcome measures beyond performance metrics, including patient-centered outcome measures. Our explanations aim to guide clinicians in the appropriate use of these metrics and measures.

Comparison of clinical and MRI features of hepatic angiosarcoma and epithelioid hemangioendothelioma.

OBJECTIVE: Differentiating hepatic epithelioid hemangioendothelioma (EHE) and angiosarcoma (AS), the two most common vascular tumors in the liver, is important due to disparities in their prognosis and treatment. We aimed to compare clinical and MRI features of the two tumors. METHODS: This retrospective study included patients with pathologically-confirmed AS or EHE who underwent MRI using gadoxetate disodium between 2008 and 2023. Two radiologists independently reviewed MR images. Wilcoxon rank sum and Fisher's exact tests were used to compare clinical and imaging features. Overall survival was compared using restricted mean survival time at 3 years. RESULTS: 32 patients with AS (18 women [56.3%]; median age, 68 years) and 38 with EHE (24 women [63.2%]; 51 years) were included. Patients with AS were generally older (81.3% ≥ 60 years; P < 0.001), had more frequent laboratory abnormalities (P ≤ 0.018), and poorer overall survival (11.2 vs. 31.8 months; P < 0.001) than those with EHE. On MRI, a large dominant mass accompanied by smaller nodules (14/32, 43.8%), often with ill-defined margins (15/32, 46.9%) was prevalent in AS; compared with nodules of similar sizes (24/38, 63.2%; P = 0.015) with well-defined margin (30/38, 78.9%; P = 0.002) in EHE. Cirrhotic appearance of the liver was more frequent in AS (62.5%, P < 0.001), along with decreased parenchymal enhancement on hepatobiliary phase (31.3%, P < 0.001) and ascites (37.5%, P = 0.010). AS frequently presented with avid enhancement of bizarrely-shaped foci, with a centrifugal enhancement pattern. In comparison, targetoid appearance was characteristic of EHE (78.9% on T2-weighted, 54.1% on diffusion-weighted, 65.8% on multiphase images) (P ≤ 0.002), with enhancement degree typically lower than that of the aorta. On hepatobiliary phase, all the AS exhibited hypointensity, while 39.5% of EHE showed targetoid appearance (P < 0.001). CONCLUSION: Patients aged ≥ 60 years presenting with laboratory abnormalities, typically with a large dominant mass accompanied by smaller nodules, exhibiting avid, bizarre, and centrifugal enhancement-particularly in the cirrhotic-appearing liver-suggests the likelihood of AS over EHE.

The Role of Imaging for Gastrointestinal Bleeding: Consensus Recommendations From the American College of Gastroenterology and Society of Abdominal Radiology.

Gastrointestinal (GI) bleeding is the most common GI diagnosis leading to hospitalization within the United States. Prompt diagnosis and treatment of GI bleeding is critical to improving patient outcomes and reducing high healthcare utilization and costs. Radiologic techniques including computed tomography angiography, catheter angiography, computed tomography enterography, magnetic resonance enterography, nuclear medicine red blood cell scan, and technetium-99m pertechnetate scintigraphy (Meckel scan) are frequently used to evaluate patients with GI bleeding and are complementary to GI endoscopy. However, multiple management guidelines exist which differ in the recommended utilization of these radiologic examinations. This variability can lead to confusion as to how these tests should be used in the evaluation of GI bleeding. In this document, a panel of experts from the American College of Gastroenterology and Society of Abdominal Radiology provide a review of the radiologic examinations used to evaluate for GI bleeding including nomenclature, technique, performance, advantages, and limitations. A comparison of advantages and limitations relative to endoscopic examinations is also included. Finally, consensus statements and recommendations on technical parameters and utilization of radiologic techniques for GI bleeding are provided.

Survey on Value Elements Provided by Artificial Intelligence and Their Eligibility for Insurance Coverage With an Emphasis on Patient-Centered Outcomes.

OBJECTIVE: This study aims to explore the opinions on the insurance coverage of artificial intelligence (AI), as categorized based on the distinct value elements offered by AI, with a specific focus on patient-centered outcomes (PCOs). PCOs are distinguished from traditional clinical outcomes and focus on patient-reported experiences and values such as quality of life, functionality, well-being, physical or emotional status, and convenience. MATERIALS AND METHODS: We classified the value elements provided by AI into four dimensions: clinical outcomes, economic aspects, organizational aspects, and non-clinical PCOs. The survey comprised three sections: 1) experiences with PCOs in evaluating AI, 2) opinions on the coverage of AI by the National Health Insurance of the Republic of Korea when AI demonstrated benefits across the four value elements, and 3) respondent characteristics. The opinions regarding AI insurance coverage were assessed dichotomously and semi-quantitatively: non-approval (0) vs. approval (on a 1-10 weight scale, with 10 indicating the strongest approval). The survey was conducted from July 4 to 26, 2023, using a web-based method. Responses to PCOs and other value elements were compared. RESULTS: Among 200 respondents, 44 (22%) were patients/patient representatives, 64 (32%) were industry/developers, 60 (30%) were medical practitioners/doctors, and 32 (16%) were government health personnel. The level of experience with PCOs regarding AI was low, with only 7% (14/200) having direct experience and 10% (20/200) having any experience (either direct or indirect). The approval rate for insurance coverage for PCOs was 74% (148/200), significantly lower than the corresponding rates for other value elements (82.5%-93.5%; P ≤ 0.034). The approval strength was significantly lower for PCOs, with a mean weight ± standard deviation of 5.1 ± 3.5, compared to other value elements (P ≤ 0.036). CONCLUSION: There is currently limited demand for insurance coverage for AI that demonstrates benefits in terms of non-clinical PCOs.

The Role of Imaging for GI Bleeding: ACG and SAR Consensus Recommendations.

Gastrointestinal (GI) bleeding is the most common GI diagnosis leading to hospitalization within the United States. Prompt diagnosis and treatment of GI bleeding is critical to improving patient outcomes and reducing high health care utilization and costs. Radiologic techniques including CT angiography, catheter angiography, CT enterography, MR enterography, nuclear medicine red blood cell scan, and technetium-99m pertechnetate scintigraphy (Meckel scan) are frequently used to evaluate patients with GI bleeding and are complementary to GI endoscopy. However, multiple management guidelines exist, which differ in the recommended utilization of these radiologic examinations. This variability can lead to confusion as to how these tests should be used in the evaluation of GI bleeding. In this document, a panel of experts from the American College of Gastroenterology and Society of Abdominal Radiology provide a review of the radiologic examinations used to evaluate for GI bleeding including nomenclature, technique, performance, advantages, and limitations. A comparison of advantages and limitations relative to endoscopic examinations is also included. Finally, consensus statements and recommendations on technical parameters and utilization of radiologic techniques for GI bleeding are provided. © Radiological Society of North America and the American College of Gastroenterology, 2024. Supplemental material is available for this article. This article is being published concurrently in American Journal of Gastroenterology and Radiology. The articles are identical except for minor stylistic and spelling differences in keeping with each journal's style. Citations from either journal can be used when citing this article. See also the editorial by Lockhart in this issue.

Interpretation of Complete Tumor Response on MRI Following Chemoradiotherapy of Rectal Cancer: Inter-Reader Agreement and Associated Factors in Multi-Center Clinical Practice.

OBJECTIVE: To measure inter-reader agreement and identify associated factors in interpreting complete response (CR) on magnetic resonance imaging (MRI) following chemoradiotherapy (CRT) for rectal cancer. MATERIALS AND METHODS: This retrospective study involved 10 readers from seven hospitals with experience of 80-10210 cases, and 149 patients who underwent surgery after CRT for rectal cancer. Using MRI-based tumor regression grading (mrTRG) and methods employed in daily practice, the readers independently assessed mrTRG, CR on T2-weighted images (T2WI) denoted as mrCRT2W, and CR on all images including diffusion-weighted images (DWI) denoted as mrCRoverall. The readers described their interpretation patterns and how they utilized DWI. Inter-reader agreement was measured using multi-rater kappa, and associated factors were analyzed using multivariable regression. Correlation between sensitivity and specificity of each reader was analyzed using Spearman coefficient. RESULTS: The mrCRT2W and mrCRoverall rates varied widely among the readers, ranging 18.8%-40.3% and 18.1%-34.9%, respectively. Nine readers used DWI as a supplement sequence, which modified interpretations on T2WI in 2.7% of cases (36/1341 [149 patients × 9 readers]) and mostly (33/36) changed mrCRT2W to non-mrCRoverall. The kappa values for mrTRG, mrCRT2W, and mrCRoverall were 0.56 (95% confidence interval: 0.49, 0.62), 0.55 (0.52, 0.57), and 0.54 (0.51, 0.57), respectively. No use of rectal gel, larger initial tumor size, and higher initial cT stage exhibited significant association with a higher inter-reader agreement for assessing mrCRoverall (P ≤ 0.042). Strong negative correlations were observed between the sensitivity and specificity of individual readers (coefficient, -0.718 to -0.963; P ≤ 0.019). CONCLUSION: Inter-reader agreement was moderate for assessing CR on post-CRT MRI. Readers' varying standards on MRI interpretation (i.e., threshold effect), along with the use of rectal gel, initial tumor size, and initial cT stage, were significant factors associated with inter-reader agreement.

CT Colonography Reporting and Data System (C-RADS): Version 2023 Update.

The CT Colonography Reporting and Data System (C-RADS) has withstood the test of time and proven to be a robust classification scheme for CT colonography (CTC) findings. C-RADS version 2023 represents an update on the scheme used for colorectal and extracolonic findings at CTC. The update provides useful insights gained since the implementation of the original system in 2005. Increased experience has demonstrated confusion on how to classify the mass-like appearance of the colon consisting of soft tissue attenuation that occurs in segments with acute or chronic diverticulitis. Therefore, the update introduces a new subcategory, C2b, specifically for mass-like diverticular strictures, which are likely benign. Additionally, the update simplifies extracolonic classification by combining E1 and E2 categories into an updated extracolonic category of E1/E2 since, irrespective of whether a finding is considered a normal variant (category E1) or an otherwise clinically unimportant finding (category E2), no additional follow-up is required. This simplifies and streamlines the classification into one category, which results in the same management recommendation.

Radiofrequency ablation of subcapsular versus nonsubcapsular hepatocellular carcinomas ≤ 3 cm: analysis of long-term outcomes from two large-volume liver centers.

OBJECTIVES: To compare the safety and efficacy of RFA for single HCCs ≤ 3 cm in subcapsular versus nonsubcapsular locations using a propensity score matched analysis. MATERIALS AND METHODS: This retrospective study included patients with solitary HCCs ≤ 3 cm in size who underwent percutaneous RFA from 2005 to 2015 as initial treatment at two large-volume liver centers. Patients were divided into two groups, consisting of those with subcapsular and nonsubcapsular tumor locations. Complications, local tumor progression (LTP), and overall survival (OS) were compared in these two groups before and after propensity score matching (PSM). RESULTS: The study population consisted of 964 patients (712 men [74%]) of mean age 58.3 years. Of these 964 patients, 561 (58%) had nonsubcapsular and 403 (42%) had subcapsular HCCs. PSM generated 402 pairs of patients. Major complication rate was low, but significantly higher in the subcapscular group (p = 0.047). Rates of technical effectiveness in these two groups were 99% and 98%, respectively (p = 0.315). However, during follow-up, cumulative 1-, 3-, 5-, and 10-year LTP and OS rates did significantly differ in both entire and PSM cohorts, resulting in the latter 8%, 15%, 20%, and 26% in the nonsubcapsular group vs. 13%, 24%, 30%, and 31% in the subcapsular group (p = 0.015), and 99%, 91%, 80%, and 59% vs. 98%, 85%, 73%, and 50% in the two groups (p = 0.004), respectively. CONCLUSION: Rates of major complications, LTP, and OS differed significantly following first-line RFA treatment of single HCCs ≤ 3 cm in favor of the nonsubcapsular locations. CLINICAL RELEVANCE STATEMENT: This large-scale study provides evidence that radiofrequency ablation for small (≤ 3 cm) hepatocellular carcinomas is safer and more effective in nonsubcapsular location than in subcapsular location. KEY POINTS: • There exist conflicting outcomes on the effectiveness of RFA for early HCC depending on tumor location. • Rate of local tumor progression was significantly higher in the subcapsular hepatocellular carcinomas. • Overall survival rate was significantly poorer in the subcapsular hepatocellular carcinomas.

Radiologist's Guide to Evaluating Publications of Clinical Research on AI: How We Do It.

Literacy in research studies of artificial intelligence (AI) has become an important skill for radiologists. It is required to make a proper assessment of the validity, reproducibility, and clinical applicability of AI studies. However, AI studies are generally perceived to be more difficult for clinician readers to evaluate than traditional clinical research studies. This special report-as an effective, concise guide for readers-aims to assist clinical radiologists in critically evaluating different types of clinical research articles involving AI. It does not intend to be a comprehensive checklist or methodological summary for complete clinical evaluation of AI or a reporting guideline. Ten key items for readers to check are described, regarding study purpose, function and clinical context of AI, training data, data preprocessing, AI modeling techniques, test data, AI performance, helpfulness and value of AI, interpretability of AI, and code sharing. The important aspects of each item are explained for readers to consider when reading publications on AI clinical research. Evaluating each item can help radiologists assess the validity, reproducibility, and clinical applicability of clinical research articles involving AI.