Understanding Reader Variability: A 25-Radiologist Study on Liver Metastasis Detection at CT.

Background Substantial interreader variability exists for common tasks in CT imaging, such as detection of hepatic metastases. This variability can undermine patient care by leading to misdiagnosis. Purpose To determine the impact of interreader variability associated with (a) reader experience, (b) image navigation patterns (eg, eye movements, workstation interactions), and (c) eye gaze time at missed liver metastases on contrast-enhanced abdominal CT images. Materials and Methods In a single-center prospective observational trial at an academic institution between December 2020 and February 2021, readers were recruited to examine 40 contrast-enhanced abdominal CT studies (eight normal, 32 containing 91 liver metastases). Readers circumscribed hepatic metastases and reported confidence. The workstation tracked image navigation and eye movements. Performance was quantified by using the area under the jackknife alternative free-response receiver operator characteristic (JAFROC-1) curve and per-metastasis sensitivity and was associated with reader experience and image navigation variables. Differences in area under JAFROC curve were assessed with the Kruskal-Wallis test followed by the Dunn test, and effects of image navigation were assessed by using the Wilcoxon signed-rank test. Results Twenty-five readers (median age, 38 years; IQR, 31-45 years; 19 men) were recruited and included nine subspecialized abdominal radiologists, five nonabdominal staff radiologists, and 11 senior residents or fellows. Reader experience explained differences in area under the JAFROC curve, with abdominal radiologists demonstrating greater area under the JAFROC curve (mean, 0.77; 95% CI: 0.75, 0.79) than trainees (mean, 0.71; 95% CI: 0.69, 0.73) (P = .02) or nonabdominal subspecialists (mean, 0.69; 95% CI: 0.60, 0.78) (P = .03). Sensitivity was similar within the reader experience groups (P = .96). Image navigation variables that were associated with higher sensitivity included longer interpretation time (P = .003) and greater use of coronal images (P < .001). The eye gaze time was at least 0.5 and 2.0 seconds for 71% (266 of 377) and 40% (149 of 377) of missed metastases, respectively. Conclusion Abdominal radiologists demonstrated better discrimination for the detection of liver metastases on abdominal contrast-enhanced CT images. Missed metastases frequently received at least a brief eye gaze. Higher sensitivity was associated with longer interpretation time and greater use of liver display windows and coronal images. © RSNA, 2022 Online supplemental material is available for this article.

Targeted Training Reduces Search Errors but Not Classification Errors for Hepatic Metastasis Detection at Contrast-Enhanced CT.

RATIONALE AND OBJECTIVES: Methods are needed to improve the detection of hepatic metastases. Errors occur in both lesion detection (search) and decisions of benign versus malignant (classification). Our purpose was to evaluate a training program to reduce search errors and classification errors in the detection of hepatic metastases in contrast-enhanced abdominal computed tomography (CT). MATERIALS AND METHODS: After Institutional Review Board approval, we conducted a single-group prospective pretest-posttest study. Pretest and posttest were identical and consisted of interpreting 40 contrast-enhanced abdominal CT exams containing 91 liver metastases under eye tracking. Between pretest and posttest, readers completed search training with eye-tracker feedback and coaching to increase interpretation time, use liver windows, and use coronal reformations. They also completed classification training with part-task practice, rating lesions as benign or malignant. The primary outcome was metastases missed due to search errors (<2 seconds gaze under eye tracker) and classification errors (>2 seconds). Jackknife free-response receiver operator characteristic (JAFROC) analysis was also conducted. RESULTS: A total of 31 radiologist readers (8 abdominal subspecialists, 8 nonabdominal subspecialists, 15 senior residents/fellows) participated. Search errors were reduced (pretest 11%, posttest 8%, difference 3% [95% confidence interval, 0.3%-5.1%], P = .01), but there was no difference in classification errors (difference 0%, P = .97) or in JAFROC figure of merit (difference -0.01, P = .36). In subgroup analysis, abdominal subspecialists demonstrated no evidence of change. CONCLUSION: Targeted training reduced search errors but not classification errors for the detection of hepatic metastases at contrast-enhanced abdominal CT. Improvements were not seen in all subgroups.

In Vivo Prediction of Kidney Stone Fragility Using Radiomics-Based Regression Models.

Introduction: The surgical technique for urinary stone removal is partly influenced by its fragility, as prognosticated by the clinician. This feasibility study aims to develop a linear regression model from CT-based radiomic markers to predict kidney stone comminution time in vivo with two ultrasonic lithotrites. Materials and Methods: Patients identified by urologists at our institution as eligible candidates for percutaneous nephrolithotomy were prospectively enrolled. The active engagement time of the lithotrite in breaking the stone during surgery denoted the comminution time of each stone. The comminution rate was computed as the stone volume disintegrated per minute. Stones were grouped into three fragility classes (fragile, moderate, hard), based on inverse of the comminution rates with respect to the mean. Multivariable linear regression models were trained with radiomic features extracted from clinical CT images to predict comminution times in vivo. The model with the least root mean squared error (RMSE) on comminution times and the fewest misclassification of fragility was finally selected. Results: Twenty-eight patients with 31 stones in total were included in this study. Stones in the cohort averaged 1557 (±2472) mm3 in volume and 5.3 (±7.4) minutes in comminution time. Ten stones had nonmoderate fragility. Linear regression of stone volume alone predicted comminution time with an RMSE of 6.8 minutes and missed all 10 stones with nonmoderate fragility. A fragility model that included stone volume, internal morphology, shape-based radiomics, and device type improved RMSE to below 3.3 minutes and correctly classified 20/21 moderate and 6/10 nonmoderate stones. Conclusions: CT metrics-based fragility models may provide information to surgeons regarding kidney stone fragility and facilitate the selection of stone removal procedures.

Individualized and Generalized Learner Models for Predicting Missed Hepatic Metastases.

The diagnostic performance of radiologist readers exhibits substantial variation that cannot be explained by CT acquisition protocol differences. Studying reader detectability from CT images may help identify why certain types of lesions are missed by multiple or specific readers. Ten subspecialized abdominal radiologists marked all suspected metastases in a multi-reader-multi-case study of 102 deidentified contrast-enhanced CT liver scans at multiple radiation dose levels. A reference reader marked ground truth metastatic and benign lesions with the aid of histopathology or tumor progression on later scans. Multi-slice image patches and 3D radiomic features were extracted from the CT images. We trained deep convolutional neural networks (CNN) to predict whether an average (generalized) or individual radiologist reader would detect or miss a specific metastasis from an image patch containing it. The individualized CNN showed higher performance with an area under the receiver operating characteristic curve (AUC) of 0.82 compared to a generalized one (AUC = 0.78) in predicting reader-specific detectability. Random forests were used to build the respective versions from radiomic features. Both the individualized (AUC = 0.64) and generalized (AUC = 0.59) predictors from radiomic features showed limited ability to differentiate detected from missed lesions. This shows that CNN can identify and learn automated features that are better predictors of reader detectability of lesions than radiomic features. Individualized prediction of difficult lesions may allow targeted training of idiosyncratic weaknesses but requires substantial training data for each reader.

Knowledge-Driven Interpretation of Multi-View Data in Medicine.

We propose a new approach to clinical decision support with interpretable recommendations from multi-view data. We introduce a Bayesian network structure learning method to help identify the relevant factors and their relationships. Guided by minimal domain knowledge, this method highlights the significant associations among the demography, medical and family history, lifestyle, and biomarker data to facilitate informed clinical decisions. We demonstrate the effectiveness of the method for detecting mild neurocognitive disorder in the elderly from a real-life dataset in Singapore. The empirical results show that our method achieves better interpretability in addition to comparable accuracy with respect to the benchmark studies.