Impact of and interaction between the availability of prior examinations and DBT on the interpretation of negative and benign mammograms.

RATIONALE AND OBJECTIVES: To assess the interaction between the availability of prior examinations and digital breast tomosynthesis (DBT) in decisions to recall a woman during interpretation of mammograms. MATERIALS AND METHODS: Eight radiologists independently interpreted twice 36 mammography examinations, each of which had current and prior full-field digital mammography images (FFDM) and DBT under a Health Insurance Portability and Accountability Act-compliant, institutional review board-approved protocol (written consent waived). During the first reading, three sequential ratings were provided using FFDM only, followed by FFDM + DBT, and then followed by FFDM + DBT + priors. The second reading included FFDM only, then FFDM + priors, and then FFDM + priors + DBT. Twenty-two benign cases clinically recalled, 12 negative/benign examinations (not recalled), and two verified cancer cases were included. Recall recommendations and interaction between the effect of priors and DBT on decisions were assessed (P = .05 significance level) using generalized linear model (PROC GLIMMIX, SAS, version 9.3; SAS Institute, Cary, NC) accounting for case and reader variability. RESULTS: Average recall rates in noncancer cases were significantly reduced (51%; P < .001) with the addition of DBT and with addition of priors (23%; P = .01). In absolute terms, the addition of DBT to FFDM reduced the recall rates from 0.67 to 0.42 and from 0.54 to 0.27 when DBT was available before and after priors, respectively. Recall reductions were from 0.64 to 0.54 and from 0.42 to 0.33 when priors were available before and after DBT, respectively. Regardless of the sequence in presentation, there were no statistically significant interactions between the effect of availability of DBT and priors (P = .80). CONCLUSIONS: Availability of both priors and DBT are independent primary factors in reducing recall recommendations during mammographic interpretations.

Digital breast tomosynthesis: observer performance study.

OBJECTIVE: The purpose of this study was to compare in a retrospective observer study the diagnostic performance of full-field digital mammography (FFDM) with that of digital breast tomosynthesis. MATERIALS AND METHODS: Eight experienced radiologists interpreted images from 125 selected examinations, 35 with verified findings of cancer and 90 with no finding of cancer. The four display conditions included FFDM alone, 11 low-dose projections, reconstructed digital breast tomosynthesis images, and a combined display mode of FFDM and digital breast tomosynthesis images. Observers rated examinations using the screening BI-RADS rating scale and the free-response receiver operating characteristic paradigm. Observer performance levels were measured as the proportion of examinations prompting recall of patients for further diagnostic evaluation. The results were presented in terms of true-positive fraction and false-positive fraction. Performance levels were compared among the acquisitions and reading modes. Time to view and interpret an examination also was evaluated. RESULTS: Use of the combination of digital breast tomosynthesis and FFDM was associated with 30% reduction in recall rate for cancer-free examinations that would have led to recall if FFDM had been used alone (p < 0.0001 for the participating radiologists, p = 0.047 in the context of a generalized population of radiologists). Use of digital breast tomosynthesis alone also tended to reduce recall rates, an average of 10%, although the observed decrease was not statistically significant (p = 0.09 for the participating radiologists). There was no convincing evidence that use of digital breast tomosynthesis alone or in combination with FFDM results in a substantial improvement in sensitivity. CONCLUSION: Use of digital breast tomosynthesis for breast imaging may result in a substantial decrease in recall rate.

Agreement of the order of overall performance levels under different reading paradigms.

RATIONALE AND OBJECTIVES: To investigate consistency of the orders of performance levels when interpreting mammograms under three different reading paradigms. MATERIALS AND METHODS: We performed a retrospective observer study in which nine experienced radiologists rated an enriched set of mammography examinations that they personally had read in the clinic ("individualized") mixed with a set that none of them had read in the clinic ("common set"). Examinations were interpreted under three different reading paradigms: binary using screening Breast Imaging Reporting and Data System (BI-RADS), receiver-operating characteristic (ROC), and free-response ROC (FROC). The performance in discriminating between cancer and noncancer findings under each of the paradigms was summarized using Youden's index/2+0.5 (Binary), nonparameteric area under the ROC curve (AUC), and an overall FROC index (JAFROC-2). Pearson correlation coefficients were then computed to assess consistency in the ordering of observers' performance levels. Statistical significance of the computed correlation coefficients was assessed using bootstrap confidence intervals obtained by resampling sets of examination-specific observations. RESULTS: All but one of the computed pair-wise correlation coefficients were larger than 0.66 and were significantly different from zero. The correlation between the overall performance measures under the Binary and ROC paradigms was the lowest (0.43) and was not significantly different from zero (95% confidence interval -0.078 to 0.733). CONCLUSION: The use of different evaluation paradigms in the laboratory tends to lead to consistent ordering of the overall performance levels of observers. However, one should recognize that conceptually similar performance indexes resulting from different paradigms often measure different performance characteristics and thus disagreements are not only possible but frequently quite natural.

Binary and multi-category ratings in a laboratory observer performance study: a comparison.

The authors investigated radiologists, performances during retrospective interpretation of screening mammograms when using a binary decision whether to recall a woman for additional procedures or not and compared it with their receiver operating characteristic (ROC) type performance curves using a semi-continuous rating scale. Under an Institutional Review Board approved protocol nine experienced radiologists independently rated an enriched set of 155 examinations that they had not personally read in the clinic, mixed with other enriched sets of examinations that they had individually read in the clinic, using both a screening BI-RADS rating scale (recall/not recall) and a semi-continuous ROC type rating scale (0 to 100). The vertical distance, namely the difference in sensitivity levels at the same specificity levels, between the empirical ROC curve and the binary operating point were computed for each reader. The vertical distance averaged over all readers was used to assess the proximity of the performance levels under the binary and ROC-type rating scale. There does not appear to be any systematic tendency of the readers towards a better performance when using either of the two rating approaches, namely four readers performed better using the semi-continuous rating scale, four readers performed better with the binary scale, and one reader had the point exactly on the empirical ROC curve. Only one of the nine readers had a binary "operating point" that was statistically distant from the same reader's empirical ROC curve. Reader-specific differences ranged from -0.046 to 0.128 with an average width of the corresponding 95% confidence intervals of 0.2 and p-values ranging for individual readers from 0.050 to 0.966. On average, radiologists performed similarly when using the two rating scales in that the average distance between the run in individual reader's binary operating point and their ROC curve was close to zero. The 95% confidence interval for the fixed-reader average (0.016) was (-0.0206, 0.0631) (two-sided p-value 0.35). In conclusion the authors found that in retrospective observer performance studies the use of a binary response or a semi-continuous rating scale led to consistent results in terms of performance as measured by sensitivity-specificity operating points.

The "laboratory" effect: comparing radiologists' performance and variability during prospective clinical and laboratory mammography interpretations.

PURPOSE: To compare radiologists' performance during interpretation of screening mammograms in the clinic with their performance when reading the same mammograms in a retrospective laboratory study. MATERIALS AND METHODS: This study was conducted under an institutional review board-approved, HIPAA-compliant protocol; the need for informed consent was waived. Nine experienced radiologists rated an enriched set of mammograms that they had personally read in the clinic (the "reader-specific" set) mixed with an enriched "common" set of mammograms that none of the participants had previously read in the clinic by using a screening Breast Imaging Reporting and Data System (BI-RADS) rating scale. The original clinical recommendations to recall the women for a diagnostic work-up, for both reader-specific and common sets, were compared with their recommendations during the retrospective experiment. The results are presented in terms of reader-specific and group-averaged sensitivity and specificity levels and the dispersion (spread) of reader-specific performance estimates. RESULTS: On average, the radiologists' performance was significantly better in the clinic than in the laboratory (P = .035). Interreader dispersion of the computed performance levels was significantly lower during the clinical interpretations (P < .01). CONCLUSION: Retrospective laboratory experiments may not represent either expected performance levels or interreader variability during clinical interpretations of the same set of mammograms in the clinical environment well.

A multisite telemammography system for remote management of screening mammography: an assessment of technical, operational, and clinical issues.

OBJECTIVE: This paper describes a high-quality, multisite telemammography system to enable "almost real-time" remote patient management while the patient remains in the clinic. One goal is to reduce the number of women who would physically need to return to the clinic for additional imaging procedures (termed "recall") to supplement "routine" imaging of screening mammography. MATERIALS AND METHODS: Mammography films from current and prior (when available) examinations are digitized at three remote sites and transmitted along with other pertinent information across low-level communication systems to the central site. Images are automatically cropped, wavelet compressed, and encrypted prior to transmission to the central site. At the central site, radiologists review and rate examinations on a high-resolution workstation that displays the images, computer-assisted detection results, and the technologist's communication. Intersite communication is provided instantly via a messaging "chat" window. RESULTS: The technologists recommended additional procedures at 2.7 times the actual clinical recall rate for the same cases. Using the telemammography system during a series of "off-line" clinically simulated studies, radiologists recommended additional procedures at 1.3 times the actual clinical recall rate. Percent agreement and kappa between the study and actual clinical interpretations were 66.1% and 0.315, respectively. For every physical recall potentially avoided using the telemammography system, approximately one presumed "unnecessary" imaging procedure was recommended. CONCLUSION: Remote patient management can reduce the number of women recalled by as much as 50% without performing an unreasonable number of presumed "unnecessary" procedures.