Heat maps: an aid for data analysis and understanding of ROC CAD experiments.

ABSTRACT

RATIONALE AND OBJECTIVES: To demonstrate the value of a new data visualization and exploration method for mutlireader-multicase receiver operating characteristic (MRMC-ROC) experiments of computer-aided detection (CAD) algorithms that uses three-dimensional (3D) heat maps tool adapted from gene expression array analysis. MATERIALS AND METHODS: We are using data from a clinical trial of a commercial CAD system for lung cancer detection (RapidScreen RS-2000, Riverain Medical Group, Miamisburg, OH, and Rockville, MD). 3D heat maps, originally developed for displaying changes in gene expression after cancer chemotherapy in MATLAB, were modified to display the radiologists confidence levels as they interpreted chest radiographs and used to visualize the radiologists confidence levels before and after the provision of a CAD system. RESULTS: Heat maps demonstrated the variation among radiologists in their interpretation, and the degree of variation in interpretation when a single radiologist reinterpreted the same case without and with CAD modality. They demonstrated the variability in the identification of each cancer/cancer-free case and the variability of change seen when CAD prompts were provided. CONCLUSIONS: CAD increases the consistency of interpretation of a single radiologist and of a group of radiologists. Heat maps provide a method for data visualization that clarifies the effects of reader variability in ROC CAD experiments. We demonstrated how heat maps can be used to document the complexity of reader variability and suggested how clustering can reveal both nonintuitive and intuitive groupings of cases, readers, and the interaction of both with CAD.