A constrained formulation for the receiver operating characteristic (ROC) curve based on probability summation.

We propose a principled formulation of the ROC curve that is constrained in a realistic way by the mechanism of probability summation. The constrained and conventional ROC formulations were fitted to 150 separate sets of rating data taken from previous observer studies of 250 or 529 chest radiographs. A total of 20 different readers had used either discrete or continuous rating scales to evaluate those chest cases for likelihood of separate specified abnormalities: interstitial disease, pulmonary nodule, pneumothorax, alveolar infiltrate, or rib fracture. Both ROC formulations were fitted separately to every set of rating data using maximum-likelihood statistical procedures that specified each ROC curve by normally distributed latent variables with two scaling parameters, and estimated the area below the ROC curve (Az) with its standard error. The conventional and constrained binormal formulations usually fitted ROC curves that were nearly indistinguishable in form and in Az. But when fitted to asymmetric rating data that contained few false-positive cases, the conventional ROC curves often rose steeply, then flattened and extrapolated into an unrealistic upward "hook" at the higher false-positive rates. For those sets of rating data, the constrained ROC curves (without hooks) estimated larger values for Az with smaller standard errors. The constrained ROC formulation describes observers' ratings of cases at least as well as the conventional ROC, and always guarantees a realistic fitted curve for observer performance. Its estimated parameters are easy to interpret, and may also be used to predict observer accuracy in localizing the image abnormalities.

Observer variation and the performance accuracy gained by averaging ratings of abnormality.

Six radiologists used continuous scales to rate 529 chest-film cases for likelihood of five different types of abnormalities (interstitial disease, nodule, pneumothorax, alveolar infiltrate, and rib fracture) in each of six replicated readings, yielding 36 separate ratings of each case for the five abnormalities. Separate data analyses of all cases and subsets of the difficult/subtle cases for each abnormality estimated the relative gains in accuracy (linear-scaled area below the ROC curve) obtained by averaging the case-ratings across (a) six independent replications by each reader (25% gain), (b) six different readers within each replication (34% gain), or (c) all 36 readings (48% gain). Although accuracy differed among both readers and abnormalities, ROC curves for the median ratings showed similar relative gains in accuracy, somewhat greater than those predicted from the measured rating correlations. A model for variance components in the observer's latent decision variable could predict these gains from measured correlations in the single ratings of cases. Depending on whether the model's estimates were based on realized accuracy gains or on rating correlations, about 48% or 39% of each reader's total decision variance (summed variance for positive and negative cases) consisted of random (within-reader) error that was uncorrelated between replications, another 10% or 14% came from idiosyncratic responses to individual cases, and about 43% or 47% was systematic variation that all readers found in the sampled cases.

Using localization data from image interpretations to improve estimates of performance accuracy.

A recently developed model uses the localization of abnormalities on images to improve statistical precision in measuring detection accuracy Az, the area below an observer's receiver operating characteristic (ROC) curve for ratings of sampled normal and abnormal cases. This study evaluated that improvement by investigating how much the standard error of estimated Az decreased when the statistical analysis included localization data. Comparisons of analyses with vs without localizations were made for: 1) the estimates of Az from observers' rating ROC curves for nodular lesions on clinical chest films and liver CT scans; 2) the probability of correct choices between paired samples of normal and abnormal cases (equivalent to Az); and 3) the sampling distributions of Az measured in Monte Carlo simulations of 2,000 independent rating experiments. Localization information considerably improved the precision of Az estimates, particularly when detection accuracy was low (Az approximately 0.60). These data provided roughly the same benefits in estimation precision as would two-to-fourfold increases in the sizes of both 1) the samples of positive and negative cases and 2) the observer samples used to estimate Az means.

Influence of visual distractors on detectability of liver nodules on contrast-enhanced spiral computed tomography scans.

RATIONALE AND OBJECTIVES: The authors evaluated the ability of observers to identify simulated nodules placed electronically on normal contrast material-enhanced computed tomography (CT) scans of the liver to assess the effect of nodule size and polarity on detection and localization. METHODS: Seven readers evaluated two sets of CT scans that contained 80 stimuli each. The simulated nodules were either darker or brighter than the contrast-enhanced liver and were 5.6-8.0 mm in diameter. Readers were asked to find the most suspicious-looking nodule on each section and rate the likelihood that the chosen location actually contained a nodule. RESULTS: The fraction of nodules found by each observer was substantially greater for dark nodules than for bright ones (0.679 +/- 0.03 vs 0.345 +/- 0.045, respectively [mean +/- standard error]). This difference was consistent for all nodule sizes. Additional analyses (including receiver operating characteristic curves of conditional responses) suggested that the presence of bright blood vessels distracted the readers and decreased their ability to find bright nodules. CONCLUSION: Normal vascular structures on contrast-enhanced CT scans of the liver impair an observer's ability to detect bright liver nodules.

Measuring performance efficiency and consistency in visual discriminations with noisy images.

These experiments measured the efficiency of disk discrimination performance, relative to an "ideal" observer, and compared 2 visually dissimilar tasks in which noisy image stimuli were identical for a physical calculation yielding optimum decisions. Performance consistency was measured by estimating the assumed underlying correlation in an observer's judgments about the same individual "frozen noise" images across independent replications of each condition. Larger disk sizes on the stimulus images considerably reduced observer performance efficiency (by a factor of 10) in both discrimination tasks, regardless of the image viewing distance. But even when efficiency was very low (5% or less), performance consistency still remained quite high (about 50%). About half of each observer's inefficiency appeared to reflect consistent (but suboptimal) perceptual "miscalculations" of the noisy stimulus information.

Unified measurement of observer performance in detecting and localizing target objects on images.

In this paper methods used to measure observer performance are reviewed, and a simple general model for finding and reporting target objects in gray-scale image backgrounds is presented. That model provides the basis for a combined measurement of detection and localization performance in various image-interpretation tasks, whether by human observers or by realized computer algorithms. The model assumes that (1) an observer's detection response and first choice of target location both depend on the "maximally suspicious" finding on an image, (2) a correct (first-choice) localization of the actual target occurs if and only if its location is selected as the most suspicious, and (3) a target's presence does not alter the degree of suspicion engendered by any other (normal) image findings. Formalization of these assumptions relates the ROC curve, which measures the ability to discriminate between images containing targets and images without targets, to the "Localization Response" (LROC) curve, which measures the conjoint ability to detect and correctly localize the actual targets in those images. A maximum-likelihood statistical procedure, developed for a two-parameter "binormal" version of this model, concurrently fits both the ROC and LROC curves from an observer's image ratings and target localizations for a set of image interpretations. The model's application is illustrated (and compared to standard ROC analysis) using sets of rating and localization data from radiologists asked to search chest films for pulmonary nodules. This model is then extended to multiple-report ("free-response") interpretations of multiple-target images, under the stringent requirement that an observer's detection capability and criterion for reporting possible targets both remain stationary across images and across the successive reports made on a given image. That extended model yields formulations and predictions for the so-called "Free-Response" (FROC) curve, and for a recently proposed "Alternative FROC" (AFROC) curve. Tests of that model's "stationarity" assumptions are illustrated using radiologists' free-search interpretations of chest films for pulmonary nodules, and they suggest that human observers may often violate those assumptions when making multiple-report interpretations of images.

Spiral CT of the chest: comparison of cine and film-based viewing.

PURPOSE: To determine radiologists' ability to find lung nodules on spiral computed tomographic (CT) scans of the chest with both rapid sequential (cine) and conventional film-based viewing. MATERIALS AND METHODS: Eight radiologists searched for lung nodules on spiral CT images (10-mm collimation, 10 mm/sec table speed) presented in two formats. Cine viewing was performed at a computer work-station; sections were viewed in 2-mm increments at frame rates up to 10 frames per second. Film-based viewing of images from a laser printer was performed with a lightbox; sections were viewed at 4-mm increments. Eight 3-5-mm-diameter simulated nodules were superimposed on each of five normal CT scans. RESULTS: Radiologists found a higher fraction of nodules with the cine presentation than with film (mean, 0.69 +/- 0.02 [standard error] versus 0.58 +/- 0.03, respectively [P = .006]). Diameter thresholds for nodule detection (50% correctly localized) were 3.3 and 3.5 mm, respectively. CONCLUSION: Cine viewing of spiral CT images of the chest improved radiologists' ability to detect nodules.

Enhanced displays of medical images: evaluation of the effectiveness of color, motion, and contour for detecting and localizing liver lesions.

RATIONALE AND OBJECTIVES: Many perceptual studies have shown that the detection of large, low-contrast targets is better either in color or in contrast-reversing presentations than in standard gray scale. We determined the value of several new display techniques for viewing liver computed tomography (CT) scans. METHODS: Eight observers (four radiologists and four nonradiologists) viewed sets of 100 liver CT images (50 with lesions and 50 without) under five display conditions on a Macintosh computer: (1) color (equiluminant color contrast); (2) color-luminance (combined luminance and chromatic contrast); (3) flicker (luminance contrast that reversed polarity at 2 Hz); (4) contour (shaded intensity mapping); and (5) control (conventional gray scale). Receiver operating characteristics (ROC) techniques were used for analysis. RESULTS: The measured ROC curve areas for the different viewing conditions were as follows: control = 0.77 +/- 0.01 (mean +/- standard error of the mean); color = 0.78 +/- 0.01; color-luminance = 0.82 +/- 0.01; flicker = 0.78 +/- 0.01; and contour = 0.76 +/- 0.01. The percentage of lesions correctly located ranged from 0.82 (color-luminance) to 0.75 (flicker). Performance under the color-luminance condition was significantly better than in the control condition (p = .01), whereas the other experimental conditions were not significantly different from the control condition (p > .21). CONCLUSION: The use of mixed color and luminance displays may have perceptual advantages for radiologists and can improve performance over that of gray-scale viewing.

Flattening of the contrast-detail curve for large lesions on liver CT images.

This study evaluated the relative roles of physical and perceptual factors in flattening the contrast-detail (CD) curve on liver CT scans. To estimate the role of physical factors, the theoretical CD curve for a calculated theoretical observer (i.e., a nonprewhitening matched filter) was predicted using the measured noise power spectrum and measured modulation transfer function of the CT system. Another theoretical CD curve was also produced from the output of the same calculated observer after taking the human visual response function (VRF) into account. Perceptual factors were evaluated by analyzing human observers' replicated ratings of the visibility of details super-imposed on liver CT scans. The CD curve for the calculated theoretical observer was below the CD curve actually measured for nine human observers and showed no flattening. With the VRF included, flattening of the theoretical CD curves was only produced by fixed image viewing distances of less than 30 cm, a reading style not employed by the human observers. Correlated ROC analysis of observers' replicated ratings indicated that while random, intraobserver variation was present, the magnitude of this so-called observer noise was insufficient to explain the flattening of CD curves. Use of narrow display windows did not eliminate this flattening effect. The main reason for human observers' inefficient detection of large, low contrast liver lesions appears to be a consistent misuse of the image information.

Contrast-detail curves for liver CT.

Contrast-detail curves were constructed for liver computed tomographic (CT) images using an objective method. Stimuli were created by superimposing disks at specified locations on sets of 92 normal liver CT images. Bright and dark disks of 9 sizes and 36 possible image contrasts were used. Sets of 92 stimuli were rendered on film at five window widths (64, 128, 256, 512, and 1024 HU). The contrast-detail (CD) curve flattened substantially for disks larger than 7-mm diameter, and its slope (on a log-log plot) was less than predicted from signal-detection theory. Manipulation of display window manipulation had little impact on this disks' visibility. The results indicate that human observers have difficulty visualizing large, low-contrast details on liver CT scans, and suggest that narrowing the display window will have little effect on this limitation.

Practices and attitudes about cathode-ray tube-based and film-based image interpretation.

A questionnaire was mailed to 708 practicing radiologists and 348 members of the Society for Computer applications in Radiology (SCAR) in order to evaluate current practices and attitudes regarding the perceived advantages or disadvantages of film- and CRT-based image interpretation. A total of 27% of the 1,056 questionnaires (137 practicing radiologists; 145 SCAR members were returned. Ninety percent of practicing radiologists used film at least 75% of the time. Advantages of film-based reading listed by more than 75% of the respondents included: film reading is faster, and facilitates viewing multiple images. Advantages of CRT-based reading included: access to the entire dynamic range and potential imaging processing. Desirable attributes of existing displays included: adjustable grey scale, magnification, ability to view multiple images, allow quick review, and viewing by several individuals. Valued potential advances included: multiple higher resolution monitors, image processing and multimodality display. Practicing radiologists and computer applications society members had similar attitudes. Film-based reading is still nearly universal, but radiologists are interested in CRT-based reading if such devices have the proper features and become more available.

Current use of screening laboratory tests before abdominal interventions: a survey of 603 radiologists.

A survey of 2,153 radiologists was conducted to assess both their current practices of evaluating hemostatic function and their use of blood tests before performing image-guided nonvascular abdominal interventions. Among the 603 (28%) who responded, more radiologists routinely perform prothrombin time (81%) or partial thromboplastin time (78%) tests than platelet counts (59%), and relatively few (7%) obtain bleeding times. The most common practice (51%) is to order all of the first three tests. Use of laboratory tests is quite common (greater than 75%) before biopsy of splenic masses, hemangiomas, or hepatomas and before all catheter insertions. These tests are used less frequently (less than or equal to 70%) before fine-needle procedures, including biopsy and cyst aspiration. Only one-third of the radiologists alter their evaluation in patients who have taken aspirin. Most respondents (64%) believe that there should be written guidelines on how to evaluate patients before interventional procedures. Virtually all (97%) thought such evaluation should be the radiologist's responsibility.

Visualization and detection-localization on computed tomographic images.

These studies investigated observers' ability to detect and locate highly visible liver lesions on computed tomographic (CT) images, manipulating both the lesion's location and polarity (brighter or darker than liver background). Visibility of a lesion is not sufficient to guarantee accurate localization. With clinical images, possible confusions between a lesion and coexisting normal structures (like blood vessels) is a serious constraint on observer performance.

Search and nonsearch protocols for radiographic consultation.

Six radiologists, acting as radiograph reviewers, used two different consultation protocols to differentiate among 292 ambiguous chest radiographic findings: 120 simulated nodules and 172 normal findings (previous readers' false-positive reports of nodules). The nonsearch protocol identified each finding (by film location), and reviewers rated the likelihood of each finding's being a pulmonary nodule. The search protocol asked reviewers to report and rate all locations regarded as possible nodules on each radiograph and assigned a default negative rating to any unreported finding (nodule or normal structure). Receiver operating characteristic analyses demonstrated a significantly higher accuracy for each reviewer's search-protocol discriminations between these nodules and ambiguous normal findings. This superiority-of-search result suggests that radiologists' second opinions about suspected lesions might be more accurate when consultants follow a search protocol, independently reviewing radiographs without prior knowledge of the specific findings that concerned the primary radiograph readers.

Size discrimination in computed tomographic images. Effects of feature contrast and display window.

Studies show that features on computed tomographic (CT) images in clinical formats become less detectable when the images are produced with wider CT display windows. We studied the effects of feature contrast and the display window on observer performance in higher-order tasks that involved discriminating small size differences between features on CT images. The features to be discriminated were pairs of disks (9.0 or 9.5 mm in diameter) superimposed on CT images of water phantoms. Sets of image stimuli for two different types of size-discrimination tasks were generated with various CT contrasts specified for the superimposed features and were produced on film transparencies with display windows ranging from 90 to 2880 Hounsfield units (HU) in width. Observers' performance improved with increasing CT contrast in both size discrimination tasks. Unlike performance in feature-detection tasks, however, size discrimination was unaffected by changing the CT display window over a factor of 16 (from 90 to 1440 HU). Performance fell only at the widest display window (2880 HU), for which CT noise was essentially invisible. These results suggest that the effect of changing the CT display window may depend on the spatial frequency content of image information required for a given task.

Display thresholding of images and observer detection performance.

In this experiment we studied how thresholding by the gray-scale display transformation altered the observer's ability to detect small, high-contrast lesions in both smooth (lower-noise) and sharp (higher-noise) reconstructions of computed tomographic (CT) images. Different display conditions manipulated the level setting of a narrow CT display window. This varied the fraction of thresholded pixels, set to the maximum or minimum values of brightness in relevant locations of the displayed images. Observers' ability to detect lesions in the smooth CT images was unaffected by moderate asymmetries in the display level but then abruptly declined at both the extremely high- and low-level settings. In sharp CT images, the lesion gradually decreased in detectability as the level setting became more asymmetric. These effects of the display manipulations were modeled by a realizable physical detector, whose decision variable was the cross correlation of the pixel display values with the lesion's expected profile at the specified (lesion and nonlesion) locations in the CT images. This cross-correlator model closely predicted the patterns of changes in lesion detectability that were measured for the human observers. It also predicted how thresholding of the higher or the lower CT numbers differentially altered the shapes of the observers' measured receiver-operating-characteristic curves for the smooth CT images.

Detection of small focal lesions in CT images: effects of reconstruction filters and visual display windows.

The detectability of small, high-contrast lesions was measured on CT images, simulations of those obtained by the EMI Mark I scanner. Images were reconstructed using five reconstruction filters (kernels), which varied the image sharpness and noise level. Different sets of images were produced using various CT display windows, six different window sizes and four different display level settings. The measured lesion detectability for observers increased from 1.6 to 2.4 as the reconstruction kernel became smoother, and it decreased only slightly at the largest display window (1000 CT numbers wide). These effects were predicted by changes in the signal-to-noise ratio, as calculated for the lesion-matched filter applied to each set of physical CT images. This filter computes the cross-correlation of the CT image and the lesion profile at the specified possible locations for the lesion.

The value of searching films without specific preconceptions.

The detection of simulated pulmonary nodules was measured in three different situations that required: (1) reports of all significant chest findings (Free Search), (2) only reports of possible nodules (Nodule Search), or (3) evaluation of particular film locations as nodules (Specified Location). ROC curves from these conditions compared how accurately readers could distinguish between the nodule and normal "test locations," and between the films that did and did not contain nodules. In replication of previous results for heterogeneous chest findings, detection accuracy was superior when readers had to search the films. Each reader's distinctions between the nodule and normal test locations became considerably less accurate when these locations were prespecified for explicit evaluation. When asked to search only for pulmonary nodules, most readers substantially increased their false reports of nodules at each level of confidence, with little improvement in their (already high) true-detection rates. Searching for nodules increased both the true-detection and false-detection rates for two readers, who simply appeared to relax a strong bias against reporting nodules in the initial Free-Search situation.