Measuring the accuracy of diagnostic systems.

Diagnostic systems of several kinds are used to distinguish between two classes of events, essentially "signals" and "noise". For them, analysis in terms of the "relative operating characteristic" of signal detection theory provides a precise and valid measure of diagnostic accuracy. It is the only measure available that is uninfluenced by decision biases and prior probabilities, and it places the performances of diverse systems on a common, easily interpreted scale. Representative values of this measure are reported here for systems in medical imaging, materials testing, weather forecasting, information retrieval, polygraph lie detection, and aptitude testing. Though the measure itself is sound, the values obtained from tests of diagnostic systems often require qualification because the test data on which they are based are of unsure quality. A common set of problems in testing is faced in all fields. How well these problems are handled, or can be handled in a given field, determines the degree of confidence that can be placed in a measured value of accuracy. Some fields fare much better than others.

The Relative Operating Characteristic in Psychology: A technique for isolating effects of response bias finds wide use in the study of perception and cognition.

The clinician looking, listening, or feeling for signs of a disease may far prefer a false alarm to a miss, particularly if the disease is serious and contagious. On the other hand, he may believe that the available therapy is marginally effective, expensive, and debilitating. The pilot seeing the landing lights only when they are a few yards away may decide that his plane is adequately aligned with the runway if he is alone and familiar with that plight. He may be more inclined to circle the field before another try at landing if he has many passengers and recent memory of another plane crashing under those circumstances. The Food and Drug administrator suspecting botulism in a canned food may not want to accept even a remote threat to the public health. But he may be less clearly biased if a recent false alarm has cost a canning company millions of dollars and left some damaged reputations. The making of almost any fine discrimination is beset with such considerations of probability and utility, which are extraneous and potentially confounding when one is attempting to measure the acuity of discrimination per se. The ROC is an analytical technique, with origins in statistical decision theory and electronic detection theory, that quite effectively isolates the effects of the observer's response bias, or decision criterion, in the study of discrimination behavior. This capability, pursued through a century of psychological testing, provides a relatively pure measure of the discriminability of different stimuli and of the capacity of organisms to discriminate. The ROC also treats quantitatively the response, or decision, aspects of choice behavior. The decision parameter can then be functionally related to the probabilities of the stimulus alternatives and to the utilities of the various stimulus-response pairs, or to the observer's expectations and motivations. In separating and quantifying discrimination and decision processes, the ROC promises a more reliable and valid solution to some practical problems and enhances our understanding of the perceptual and cognitive phenomena that depend directly on these fundamental processes. In several problem areas in psychology, effects that were supposed to reflect properties of the discrimination process have been shown by the ROC analysis to reflect instead properties of the decision process.

Assessment of diagnostic technologies.

A general protocol for rigorous evaluation of diagnostic systems in medicine was applied successfully in a comparative study of two radiologic techniques. Accuracies of computed tomography and radionuclide scanning in detecting, localizing, and diagnosing brain lesions were assessed with a sample of patients in whom tumor had been suspected. The principal means of analysis was the "relative operating characteristic," which is unique in providing a measure of accuracy that is largely independent of decision biases. Computed tomography was found to be substantially more accurate than radionuclide scanning.

ROC analysis applied to the evaluation of medical imaging techniques.

Analysis in terms of the relative operating characteristic (ROC) has recently been applied to several studies of medical decision-making, primarily to decisions based on imaging techniques. This paper presents a brief description of the ROC, and shows how it provides a measure of diagnostic accuracy that is free of judgmental bias. The results of medical studies are reviewed, and the main questions of theory and method that have arisen in the medical context are identified. Certain of these questions are basic to any psychophysical test, in which case an attempt has been made to present the best available answers. Other questions are of special medical importance and relevant reports are reviewed along with a description of current efforts to provide answers.

Enhanced interpretation of diagnostic images.

In radiology, as in various other fields, observers study images to detect and diagnose underlying conditions. They make assessments of several image features and merge them into an overall decision. Demonstration is given here, in the context of mammography, that objective aids to this interpretative process can substantially improve accuracy, even for sophisticated and motivated radiologists. The aids are a checklist that solicits explicit, quantitative, systematic assessments of the important features of an image and a computer program that merges those assessments with optimal weights. The computer issues estimates of the likelihoods that specified conditions are present (in this study, the likelihood that a localized abnormality is malignant), and the radiologist benefits from taking those estimates as guidance.

The science of choosing the right decision threshold in high-stakes diagnostics.

Many diagnostic tasks require that a threshold be set to convert evidence that is a matter of degree into a positive or negative decision. Although techniques of decision analysis used in psychology help one select the particular threshold that is appropriate to a given situation and purpose, just the concept of adjusting the threshold to the situation is not appreciated in many important practical arenas. Testing for the human immunodeficiency virus (HIV) and for dangerous flaws in aircraft structures are used here as illustrations. This article briefly reviews the relevant techniques and develops those two examples with data. It suggests that use of the decision techniques could substantially benefit individuals and society and asks how that use might be facilitated.

Enhancing and evaluating diagnostic accuracy.

Techniques that may enhance diagnostic accuracy in clinical settings were tested in the context of mammography. Statistical information about the relevant features among those visible in a mammogram and about their relative importances in the diagnosis of breast cancer was the basis of two decision aids for radiologists: a checklist that guides the radiologist in assigning a scale value to each significant feature of the images of a particular case, and a computer program that merges those scale values optimally to estimate a probability of malignancy. A test set of approximately 150 proven cases (including normals and benign and malignant lesions) was interpreted by six radiologists, first in their usual manner and later with the decision aids. The enhancing effect of these feature-analytic techniques was analyzed across subsets of cases that were restricted progressively to more and more difficult cases, where difficulty was defined in terms of the radiologists' judgements in the standard reading condition. Accuracy in both standard and enhanced conditions decreased regularly and substantially as case difficulty increased, but differentially, such that the enhancement effect grew regularly and substantially. For the most difficult case sets, the observed increases in accuracy translated into an increase of about 0.15 in sensitivity (true-positive proportion) for a selected specificity (true-negative proportion) of 0.85 or a similar increase in specificity for a selected sensitivity of 0.85. That measured accuracy can depend on case-set difficulty to different degrees for two diagnostic approaches has general implications for evaluation in clinical medicine. Comparative, as well as absolute, assessments of diagnostic performances--for example, of alternative imaging techniques--may be distorted by inadequate treatments of this experimental variable. Subset analysis, as defined and illustrated here, can be useful in alleviating the problem.

Patient-oriented performance measures of diagnostic tests. 2. Assignment potential and assignment strength.

Assignment Potential (AP) is a performance measure of a diagnostic test, characterizing the chance that, as a consequence of performing the test, the probability of disease will exceed a decision threshold, thereby permitting a management action to be taken. Another performance measure, Assignment Strength (AS) characterizes the average extent to which a decision threshold will be exceeded when the post-test probability of disease does exceed the threshold. Both AP and AS are functions of prior probability of disease and decision threshold, and can be represented as two-dimensional contour maps indicating their behavior throughout the entire probability and threshold space. AP and AS can be determined for both discrete-valued tests and tests with continuous spectra of results. The contour map displays facilitate determination of the values of these measures at any prior probability and threshold, as well as visual sensitivity analysis for ranges of prior probability and/or threshold. AP and AS may be useful to the clinician in prospective evaluation of a diagnostic test in situations where formal decision analysis is not feasible.

Combining evidence from multiple imaging modalities: a feature-analysis method.

This study was designed to develop methods to improve radiologists' ability to detect and diagnose breast cancer. We evaluated the ability of a feature-analysis method to help radiologists merge judgements constructively from two rather disparate breast imaging tests. To accomplish these goals, we developed a list of perceptual features and quantitated the importance of each in the diagnosis of patients having both diaphanography (Test 1) and mammography (Test 2). Then, two decision aids were developed: One was a checklist of the critical diagnostic visual features from both tests that also assisted readers in rating these features numerically. The second was a computer-based classifier that assisted readers in merging the assessments of the two tests into one overall diagnostic probability. The value of these aids was assessed by comparing radiologists' accuracy in reading a set of proven cases in their standard fashion with their accuracy when reading in an enhanced mode, utilizing the checklist and computer classifier. When Test 1 was read adjunctively with Test 2, use of the decision aids led to a significant improvement in accuracy (p = .013) over the unenhanced, combined readings. For Test 1 alone, the aids led to a significant improvement over its low level of unenhanced reading (p = .046). For Test 2 alone, the enhancements provided little gain in accuracy over an already high level of performance on the full case set (p = .081), although significant gains were realized on the most difficult ones. We conclude that methods to aid standardization and merging of feature-based judgements can improve radiologists performance on complex diagnostic tasks.

Prostate cancer: relative effects of demographic, clinical, histologic, and MR imaging variables on the accuracy of staging.

PURPOSE: To determine the effects on the accuracy of staging prostate gland cancer of diagnostic prediction rules based on demographic, clinical, histologic, and magnetic resonance (MR) image variables. MATERIALS AND METHODS: A total of 200 cases from four medical centers were evaluated by nine radiologists experienced in MR imaging. The accuracies of the four diagnostic variables (age, prostate specific antigen level, Gleason tumor grade, and MR imaging findings) were measured, both singly and combined in a particular sequence, by calculating the area index of the receiver operating characteristic curve. RESULTS: The accuracy of staging with single variables (age, 0.58; prostate specific antigen level, 0.74; Gleason grade 0.73, MR image findings, 0.74) increased as the variables were optimally merged. The first two variables combined to yield an accuracy of 0.74; the first three combined to yield an accuracy of 0.81; and all four variables resulted in an accuracy of 0.86. In a clinically important subset of 69 cases for which antigen level and Gleason grade together were inconclusive for the purposes of staging, the addition of MR imaging findings resulted in an increase in accuracy from 0.55 to 0.73. CONCLUSION: Optimal merging of diagnostic test results yields an improvement in the accuracy of prostate cancer staging.

Reading and decision aids for improved accuracy and standardization of mammographic diagnosis.

Image-reading and decision aids were designed to improve the accuracy of mammogram interpretation. The reading aid was a list of diagnostic radiographic features and scales for quantification of each feature. The decision aid, a computer program, converted the reader's scaled values, weighted for predictive power, into an advisory estimate of the probability of malignancy. The features were identified and their importance was assigned in four steps: (a) interviews of five expert readers to establish an initial set of features, (b) perceptual tests to refine the feature set, (c) a consensus meeting to refine this set and establish nomenclature and scales, and (d) the expert's scaling of each feature in a set of 150 mammograms. Those scaled judgments were analyzed to provide the final list of features and their relative importance and to program the computer decision aid. To test the enhancement effect, six other radiologists interpreted a different set of mammograms without, and later with, the two aids. Receiver operating characteristic analysis showed a gain of approximately 0.05 in sensitivity or specificity when the other value remained at 0.85. In a subset of the more difficult cases, the enhancement effect was approximately 0.15 in either sensitivity or specificity.

Staging prostate cancer with MR imaging: a combined radiologist-computer system.

PURPOSE: To test the accuracy of a combined radiologist-computer system in the diagnosis with magnetic resonance (MR) imaging of cancer of the prostate gland. MATERIALS AND METHODS: The combined system was developed and tested by four specialists in prostate MR imaging and five radiologists expert in body MR imaging. Each group read MR images obtained in 100 proved cases of prostate cancer. The images were obtained from two sources, and all were obtained with an endorectal surface coil. Prostate MR specialists ranked imaging features of cases to develop a checklist for image interpretation. Features with greatest diagnostic value were incorporated in the combined system. Accuracy measures were derived from the area index of the receiver operating characteristic curve for the combined system and compared with those of radiologists working alone. RESULTS: Body MR radiologists had a mean baseline accuracy of 0.67; mean accuracy of their combined system was 0.80. The prostate MR specialists, when they rated the features in each case, had a mean accuracy of 0.81; the accuracy of their combined system was 0.87. CONCLUSIONS: A combined radiologist-computer system substantially improved accuracy of body MR radiologists in the diagnosis of prostate cancer. High levels of accuracy were also achieved by the system with prostate MR specialists.