Maximum likelihood estimation of dose-response parameters for therapeutic operating characteristic (TOC) analysis of carcinoma of the nasopharynx.

A Therapeutic Operating Characteristic (TOC) curve for radiation therapy plots, for all possible treatment doses, the probability of tumor ablation as a function of the probability of radiation-induced complication. Application of this analysis to actual therapeutic situations requires that dose-response curves for ablation and for complication be estimated from clinical data. We describe an approach in which "maximum likelihood estimates" of these dose-response curves are made, and we apply this approach to data collected on responses to radiotherapy for carcinoma of the nasopharynx. TOC curves constructed from the estimated dose-response curves are subject to moderately large uncertainties because of the limitations of available data. These TOC curves suggest, however, that treatment doses greater than 1800 ret may substantially increase the probability of tumor ablation with little increase in the risk of radiation-induced cervical myelopathy, especially for T1 and T2 tumors.

Receiver operating characteristic analysis of regional cerebral blood flow in Alzheimer's disease.

Receiver operating characteristic (ROC) curves were used to quantitatively assess the ability of individual detectors in a 32-detector 133Xe inhalation system to discriminate between two populations over the range of regional cerebral blood flow (rCBF) values. These populations were clinically evaluated as normal (age 63.1 +/- 13.1, n = 23) and presumed Alzheimer's disease (age 72.7 +/- 7.0, n = 82). Summary statistics showed that for homologous detectors the average value of blood flow in the normal group was greater than the flow value in the group of subjects with Alzheimer's disease. Conclusions drawn from single values of flow or mean hemispheric flow can lead to erroneous conclusions about hemisphere asymmetries. However, the dynamic relationship between the correct identifications (true positives) compared with incorrect identifications (false positives) of Alzheimer's disease at each detector varies over the range of blood flow values, and quantitative characterization of this relationship in terms of an ROC curve provides more insight into the structure of the data. Detectors approximating the speech, auditory and association cortex were most effective in discriminating between groups. Frontal detectors were marginally useful diagnostically.

Diagnostic performance of a receptor-binding radiopharmacokinetic model.

UNLABELLED: The aim of this study was to determine which measurement obtained from a radiopharmacokinetic model of a receptor-binding radiotracer provides the highest diagnostic performance for the detection of diffuse hepatocellular disease. METHODS: Twenty-seven healthy subjects and 46 patients with diffuse hepatocellular disease were studied with the receptor-binding radiopharmaceutical, 99mTc-galactosyl-neoglycoalbumin. A radiopharmacokinetic model was used to produce estimates of receptor concentration [R]o, the scaled forward-binding rate constant Kb, hepatic plasma volume, Vh, extrahepatic plasma volume, Ve and hepatic plasma flow, F. Receiver operating characteristic analysis of each model estimate was conducted. RESULTS: Receptor concentration [R]o and the metrics [R]o/tbw and kb[R]o[R]o/tbw provided the best discrimination between healthy and diseased liver. The forward-binding rate constant kb and the metrics F/Ve and Vh/tbw provided no discrimination. CONCLUSION: Based on simplicity and higher measurement precision, [R]o was selected as the most accurate index of hepatic function.

An evaluation of maximum likelihood-expectation maximization reconstruction for SPECT by ROC analysis.

A ROC study was performed in order to evaluate whether the maximum likelihood expectation maximization (ML-EM) reconstruction algorithm improves diagnostic performance compared to the conventional filtered backprojection method in SPECT. Several implementations of the algorithm were tested including 25 and 50 iteration stopping points, with and without nonuniform attenuation compensation, and with and without Metz filtering. Filtered backprojection was with Metz filter and without attenuation compensation. The test data were computer simulated to model cardiac 201Tl SPECT. The data incorporated the effects of nonuniform attenuation, distance-dependent collimator response, and scatter. Patient CT images provided realistic anatomy and attenuation information for the data simulation. Four observers each viewed 120 images for each of the reconstruction methods. Lesion detectability with ML-EM increased with Metz filtering and decreased with nonuniform attenuation compensation. The best MIL-EM implementation, 50 iterations with Metz filtering and without attenuation compensation, was not statistically better than filtered backprojection.

Effects of scatter substraction on detection and quantitation in hepatic SPECT.

UNLABELLED: The purpose of this investigation was to examine the effects of subtractive scatter compensation methods on lesion detection and quantitation. METHODS: Receiver operating characteristic (ROC) methodology was used to measure human observer detection accuracy for tumors in the liver using synthetic images. Furthermore, ROC results were compared with mathematical models for detection and activity quantitation to examine (a) the potential for predicting human performance and (b) the relationship between the detection and quantitation tasks. Images with both low and high amounts of scatter were compared with the ideal case of images of primary photons only (i.e., perfect scatter rejection) and with images corrected by subtracting a scatter image estimated by the dual photopeak window method. RESULTS: With low contrast tumors in a low count background, the results showed that scatter subtraction improved quantitation but did not produce statistically significant increases in detection accuracy. However, primary images did produce some statistically significant improvements in detection accuracy when compared with uncorrected images, particularly for high levels of scatter. CONCLUSION: Although scatter subtraction methods may provide improved activity quantitation, they may not significantly improve detection for liver SPECT. The results imply that significant improvement in detection accuracy for the conditions tested may depend on the development of gamma cameras with better scatter rejection.

An evaluation of lesion detectability with cone-beam, fanbeam and parallel-beam collimation in SPECT by continuous ROC study.

UNLABELLED: To evaluate lesion detectability for clinical evaluation of cone-beam (CB), fanbeam (FB) and parallel-beam (PB) collimator sensitivity, experimentally acquired phantom data were used to assess the advantage of CB collimation over conventional collimation. METHODS: Lesion detectability with CB, FB and PB collimation in SPECT was compared using a three-dimensional brain phantom and continuous receiver operating characteristic (CROC) analysis. A simulated cold lesion was located near the posterior portion of the thalamus. High count density scans of this phantom were acquired with CB, FB and PB collimators with similar resolution. These projections were scaled to count levels which reflected the measured sensitivities of the three collimators. Computer-generated Poisson noise was added to the projections to produce uncorrelated data sets. Images were reconstructed using a filtered backprojection algorithm. All reconstructions used a Hann filter with multiplicative attenuation correction. Each of seven trained observers viewed 288 sets of images and indicated the certainty of perceiving a cold lesion at a specified location by a rating of 0-100. Each image set contained four adjacent slices centered on the lesion to minimize partial volume effects. The program LABROC4 was used to fit CROC curves to individual observers' ratings. A t-test for paired data was performed on the individual areas. RESULTS: The average areas (standard deviations) under CROC curves for CB, FB and PB were 0.89 (0.03), 0.83 (0.05) and 0.76 (0.04), respectively. The differences of the areas were statistically significant with all two-tailed p values < 0.02. CONCLUSION: These results demonstrate that cold lesions in the posterior portion of the thalamus are best detected by images obtained using CB followed by FB and PB collimation.

Results of a clinical receiver operating characteristic study comparing filtered backprojection and maximum likelihood estimator images in FDG PET studies.

The results of a receiver operator characteristic (ROC) study comparing maximum likelihood estimator (MLE) reconstructions of human FDG PET brain scan data to filtered backprojection reconstructions of the same data are reported. The purpose of the study was to determine whether MLE reconstructions would result in higher detectability of small focal lesions introduced artificially into otherwise normal scan data. One physician assisted in defining the location and intensity of the lesions and five physicians read the final images. Data from 90 datasets were used for the study. Of those, 42 were left in their original "normal" condition and 48 were modified by added lesions. All datasets were reconstructed by the two methods and submitted to the five physicians for evaluation. The results show an increase in the area under the ROC curve from approximately 0.65 for filtered backprojection to approximately 0.71 for the maximum likelihood reconstructions for four of the five observers with good statistical significance.

Basic principles of ROC analysis.

The limitations of diagnostic "accuracy" as a measure of decision performance require introduction of the concepts of the "sensitivity" and "specificity" of a diagnostic test. These measures and the related indices, "true positive fraction" and "false positive fraction," are more meaningful than "accuracy," yet do not provide a unique description of diagnostic performance because they depend on the arbitrary selection of a decision threshold. The receiver operating characteristic (ROC) curve is shown to be a simple yet complete empirical description of this decision threshold effect, indicating all possible combinations of the relative frequencies of the various kinds of correct and incorrect decisions. Practical experimental techniques for measuring ROC curves are described, and the issues of case selection and curve-fitting are discussed briefly. Possible generalizations of conventional ROC analysis to account for decision performance in complex diagnostic tasks are indicated. ROC analysis is shown to be related in a direct and natural way to cost/benefit analysis of diagnostic decision making. The concepts of "average diagnostic cost" and "average net benefit" are developed and used to identify the optimal compromise among various kinds of diagnostic error. Finally, the way in which ROC analysis can be employed to optimize diagnostic strategies is suggested.

ROC methodology in radiologic imaging.

If the performance of a diagnostic imaging system is to be evaluated objectively and meaningfully, one must compare radiologists' image-based diagnoses with actual states of disease and health in a way that distinguishes between the inherent diagnostic capacity of the radiologists' interpretations of the images, and any tendencies to "under-read" or "over-read". ROC methodology provides the only known basis for distinguishing between these two aspects of diagnostic performance. After identifying the fundamental issues that motivate ROC analysis, this article develops ROC concepts in an intuitive way. The requirements of a valid ROC study and practical techniques for ROC data collection and data analysis are sketched briefly. A survey of the radiologic literature indicates the broad variety of evaluation studies in which ROC analysis has been employed.

Some practical issues of experimental design and data analysis in radiological ROC studies.

Receiver operating characteristic (ROC) analysis has been used in a broad variety of medical imaging studies during the past 15 years, and its advantages over more traditional measures of diagnostic performance are now clearly established. But despite the essential simplicity of the approach, workers in the field often find--sometimes only after an ROC study is under way--that a number of subtle issues related to experimental design and data analysis must be confronted in practice. Many of these issues have not been discussed in the literature in detail, and most are not well known. The purposes of this paper are to make users of ROC methodology in medical imaging aware of potential problems that should be confronted before an ROC study is begun and to indicate, at least broadly, how those problems may be dealt with, given the present state of the art. Some of the issues raised here can be addressed adequately by easily prescribed techniques, whereas others remain difficult and will be resolved fully only by new methodologic developments.

The use of continuous and discrete confidence judgments in receiver operating characteristic studies of diagnostic imaging techniques.

Virtually all receiver operating characteristic (ROC) studies of medical imaging methods have involved data collected on a discrete confidence-rating scale, though in principle ROC studies can be conducted with either discrete or continuous scales. Continuous scales provide potential advantages in medical applications of ROC analysis, but the possible dependence of ROC results on the kind of scale used in an observer-performance experiment has not been investigated empirically. The authors conducted a multi-observer ROC study in which radiologists read abdominal computed tomography (CT) examination results displayed on a workstation with a high-resolution video monitor. Ninety-five examinations were evaluated by five radiologists who reported their confidence concerning the presence of abdominal masses on a conventional five-category discrete scale and, in a separate reading session, on a continuous subjective-probability scale. ROC analysis showed no statistically significant difference between the detection accuracies (Az) achieved with the two scales by any reader. Accuracies averaged over all readers were identical with the two scales. The results of this study suggest that discrete or continuous scales often can be used interchangeably in image-evaluation studies. However, the authors recommend continuous scales for routine use in radiologic ROC studies because of their potential advantages in some situations.

Digital mammography. ROC studies of the effects of pixel size and unsharp-mask filtering on the detection of subtle microcalcifications.

We investigated the spatial resolution requirement and the effect of unsharp-mask filtering on the detectability of subtle microcalcifications in digital mammography. Digital images were obtained by digitizing conventional screen-film mammograms with a 0.1 X 0.1 mm2 pixel size, processed with unsharp masking, and then reconstituted on film with a Fuji image processing/simulation system (Fuji Photo Film Co., Tokyo, Japan). Twenty normal cases and 12 cases with subtle microcalcifications were included. Observer performance experiments were conducted to assess the detectability of subtle microcalcifications in the conventional, the unprocessed digital, and the unsharp-masked mammograms. The observer response data were evaluated using receiver operating characteristic (ROC) and LROC (ROC with localization) analyses. Our results indicate that digital mammograms obtained with 0.1 X 0.1 mm2 pixels provide lower detectability than the conventional screen-film mammograms. The detectability of microcalcifications in the digital mammograms is improved by unsharp-mask filtering; the processed mammograms still provide lower accuracy than the conventional mammograms, however, chiefly because of increased false-positive detection rates for the processed images at each subjective confidence level. Viewing unprocessed digital and unsharp-masked images in pairs resulted in approximately the same detectability as that obtained with the unsharp-masked images alone. However, this result may be influenced by the fact that the same limited viewing time was necessarily divided between the two images.

Evaluation of an enhanced digital film-duplication system by receiver operating characteristic analysis.

RATIONALE AND OBJECTIVES: Over- or underexposure often results in poor quality radiographs. Using receiver operating characteristic analysis, the authors investigated the potential utility of a system that corrects densities in such images to improve the accuracy of lung nodule detection. METHODS: Chest radiographs were digitized by a laser film digitizer. Simulated nodules were superimposed on 36 improperly exposed chest radiographs. All images, with and without nodules, were then processed by a nonlinear density-correction technique. Hard-copy images were produced by a laser film printer. Eight radiologists evaluated the corrected and uncorrected images for the presence or absence of lung nodules using a continuous (0-100) confidence-rating scale. RESULT: The detectability of lung nodules improved significantly in the retrocardiac/retrodiaphragmatic regions of underexposed images and in the peripheral lung region of overexposed images. CONCLUSIONS: This system may increase diagnostic accuracy in detection of lung nodules in improperly exposed images.

Effect of two rating formats in multi-disease ROC study of chest images.

ROC methodology has been expanded in recent years to include multi-disease experiments. To accommodate these changes, different rating formats, general or disease specific, can be used. No experimental data are available concerning the possible effects of the rating format on the results of these studies. We performed a multi-observer, multi-disease study in which 196 chest images were rated using a format where each disease was evaluated individually and one in which the cases were evaluated without scoring a specific disease. The results indicate that for our data set, the overall assessment of accuracy was not significantly affected by the study format used. Thus, in spite of the difficulties in selecting appropriate controls and the necessity of reassessing sample size considerations, the disease-specific format appears to produce an assessment of accuracy equivalent to that produced by the general format. This equivalence permits the use of the disease-specific approach since it more closely simulates the readers' true environment and is more appropriate for comparing imaging systems that may have a relative accuracy that is disease specific.

Effect of observer instruction on ROC study of chest images.

Receiver-operating characteristic (ROC) analysis has been used in many medical imaging applications during the past decade. In order to ensure that reader-confidence ratings are analyzable (well distributed to meet convergence requirements of curve-fitting algorithms) and meaningful (limit extrapolation of the data), many investigators train readers specifically for this purpose. No experimental data are available concerning the possible effects of such training on the results of ROC studies. We performed a multi-observer, multi-disease study in which 300 chest images were rated by four radiologists before and after they were trained to provide well-distributed confidence ratings. The results indicate that for our data set, reader and disease-specific accuracy was not significantly affected by the training process for interstitial disease and pneumothoraces. However, the accuracy of two readers was significantly affected for the detection of nodules (P less than 0.05), and the overall accuracy of one reader was significantly affected for the classification of normal versus abnormal images (P less than 0.01). Thus, in spite of the difficulties associated with the performance of ROC studies in a free-reading environment, one should carefully consider the possible effects of any intervention on the results prior to conducting ROC studies.

Improvement in radiologists' detection of clustered microcalcifications on mammograms. The potential of computer-aided diagnosis.

Relatively simple, but important, detection tasks in radiology are nearing accessibility to computer-aided diagnostic (CAD) methods. The authors have studied one such task, the detection of clustered microcalcifications on mammograms, to determine whether CAD can improve radiologists' performance under controlled but generally realistic circumstances. The results of their receiver operating characteristic (ROC) study show that CAD, as implemented by their computer code in its present state of development, does significantly improve radiologists' accuracy in detecting clustered microcalcifications under conditions that simulate the rapid interpretation of screening mammograms. The results suggest also that a reduction in the computer's false-positive rate will further improve radiologists' diagnostic accuracy, although the improvement falls short of statistical significance in this study.

Critical evaluation of stress testing in the diagnosis of peripheral vascular disease.

We studied 218 patients (372 limbs) and 25 normal subjects (50 limbs) with resting ankle index (RAI), treadmill exercise (TE), and postocclusive reactive hyperemia (PORH) to determine whether diagnostic accuracy is improved through the use of stress testing. In addition, we studied 10 patients with stable claudication (20 limbs) to determine the reproducibility of the three measures. RAI was the most reproducible measure, with the smallest variance between testing days (P less than 0.001). RAI differentiated between arteriographically diseased and normal limbs with a sensitivity of 97% and a specificity of 100%, whereas the corresponding values for TE were 97% and 96% and for PORH 89% and 96%. Recovery to baseline index was prolonged in the diseased group compared with normal (p less than 0.001 for both TE and PORH), but this was of limited discriminative value. Receiver-operating characteristic curve analysis documented that RAI was as diagnostically useful as TE and that both were more valuable than PORH (P less than 0.02). However, the routine addition of stress testing increased diagnostic yield by only 1.6% and cost $1100 for each limb correctly diagnosed through the addition of stress testing. RAI is a simple, accurate, and reproducible test. Routine stress testing is not cost effective, adding little diagnostic information to RAI, and it should be reserved for the small subset of symptomatic patients with normal RAI.

Clinical utility of the erythrocyte sedimentation rate in preoperative evaluation of solitary skeletal lesions.

We investigated the utility of the erythrocyte sedimentation rate in planning the diagnostic and surgical strategy for patients who were found by conventional radiography to have a solitary skeletal lesion. A series of 101 patients was separated into two groups according to their ultimate diagnoses. Group I included patients with skeletal metastases of unknown origin, unusual bone infections, and marrow cell tumors. Group II included benign and malignant primary bone tumors. Using the Westergren method to determine the erythrocyte sedimentation rate prior to biopsy, we found that only one of 65 Group II patients had a rate above 30 mm/h whereas 22 of 36 Group I patients had rates greater than 60 mm/h. By using Receiver Operating Characteristic analysis, we estimated the trade-offs between sensitivity and specificity that can be achieved with various threshold sedimentation rates. In particular, we found that a critical erythrocyte sedimentation rate of 30 mm/h allows separation of Group II patients from Group I patients with 96.9% sensitivity and 82.6% specificity. Our results indicate that the erythrocyte sedimentation rate can be used effectively in conjunction with conventional radiography in the planning of an appropriate further diagnostic strategy.

The magnetization transfer effect in cerebral infarction.

PURPOSE: To estimate the age of cerebral infarcts using magnetization transfer. METHODS: Twelve patients with radiographically and clinically documented cortical and subcortical cerebral infarctions underwent MR imaging on a 0.1T magnet. Magnetization transfer contrast images were generated by application of off-resonance pulses to every other repetition time on intermediate-weighted images. The magnetization transfer effect was calculated by obtaining an intensity value in the region of interest within the infarcted area. RESULTS: The data show a pattern in which the magnetization transfer effect decreases as the chronicity of the infarct increases. Infarcts less than 1 week old had an average magnetization transfer effect of 0.35. Those more than 1 week and less than 1 month old average 0.30. Infarcts more than 1 month and specifically those more than 1 year old averaged 0.16 or less. CONCLUSION: Magnetization transfer offers the potential to estimate the age of cerebral infarcts.

Magnetization transfer: a potential method to determine the age of multiple sclerosis lesions.

PURPOSE: To determine whether magnetization transfer contrast can differentiate acute from chronic lesions in multiple sclerosis. METHODS: Thirteen patients with multiple sclerosis and eight healthy patients were studied with MR using a 0.1-T system. Relatively T2-weighted spin-echo images were obtained without and with magnetization transfer contrast. The magnetization transfer effect of multiple sclerosis lesions was calculated and compared with the ages of the lesions. The magnetization transfer effect of normal-appearing white matter in patients with multiple sclerosis was calculated and compared with the magnetization transfer effect of white matter in healthy volunteers. Statistical analysis was performed. RESULTS: White matter in the healthy volunteers had values from 0.40 to 0.45. Normal-appearing white matter in the patients with multiple sclerosis had magnetization transfer effect values ranging from 0.41 to 0.45. Multiple sclerosis plaques of less than 1 year's duration had magnetization transfer effect values ranging from 0.05 to 0.26; older plaques had values from 0.25 to 0.41. The difference in the distributions of these values for acute and chronic multiple sclerosis plaques is statistically significant. CONCLUSION: Current imaging modalities do not differentiate acute multiple sclerosis lesions from chronic ones. Our data on magnetization transfer show a statistically significant difference in magnetization transfer effect values between lesions of less than 1 year's duration and older lesions. The different values may correspond to the histologic changes of multiple sclerosis plaques over time. Magnetization transfer may be a reliable method for determining the age of multiple sclerosis lesions.