Ideal observer approximation using Bayesian classification neural networks.

It is well understood that the optimal classification decision variable is the likelihood ratio or any monotonic transformation of the likelihood ratio. An automated classifier which maps from an input space to one of the likelihood ratio family of decision variables is an optimal classifier or "ideal observer." Artificial neural networks (ANNs) are frequently used as classifiers for many problems. In the limit of large training sample sizes, an ANN approximates a mapping function which is a monotonic transformation of the likelihood ratio, i.e., it estimates an ideal observer decision variable. A principal disadvantage of conventional ANNs is the potential over-parameterization of the mapping function which results in a poor approximation of an optimal mapping function for smaller training samples. Recently, Bayesian methods have been applied to ANNs in order to regularize training to improve the robustness of the classifier. The goal of training a Bayesian ANN with finite sample sizes is, as with unlimited data, to approximate the ideal observer. We have evaluated the accuracy of Bayesian ANN models of ideal observer decision variables as a function of the number of hidden units used, the signal-to-noise ratio of the data and the number of features or dimensionality of the data. We show that when enough training data are present, excess hidden units do not substantially degrade the accuracy of Bayesian ANNs. However, the minimum number of hidden units required to best model the optimal mapping function varies with the complexity of the data.

Components-of-variance models for random-effects ROC analysis: the case of unequal variance structures across modalities.

RATIONALE AND OBJECTIVES: Several of the authors have previously published an analysis of multiple sources of uncertainty in the receiver operating characteristic (ROC) assessment and comparison of diagnostic modalities. The analysis assumed that the components of variance were the same for the modalities under comparison. The purpose of the present work is to obtain a generalization that does not require that assumption. MATERIALS AND METHODS: The generalization is achieved by splitting three of the six components of variance in the previous model into modality-dependent contributions. Two distinct formulations of this approach can be obtained from alternative choices of the three components to be split; however, a one-to-one relationship exists between the magnitudes of the components estimated from these two formulations. RESULTS: The method is applied to a study of multiple readers, with and without the aid of a computer-assist modality. performing the task of discriminating between benign and malignant clusters of microcalcifications. Analysis according to the first method of splitting shows large decreases in the reader and reader-by-case components of variance when the computer assist is used by the readers. Analysis in terms of the alternative splitting shows large decreases in the corresponding modality-interaction components. CONCLUSION: A solution to the problem of multivariate ROC analysis without the assumption of equal variance structure across modalities has been provided. Alternative formulations lead to consistent results related by a one-to-one mapping. A surprising result is that estimates of confidence intervals and numbers of cases and readers required for a specified confidence interval remain the same in the more general model as in the restricted model.

Continuous versus categorical data for ROC analysis: some quantitative considerations.

RATIONALE AND OBJECTIVES: Several authors have encouraged the use of a quasi-continuous rating scale for data collection in receiver operating characteristic (ROC) curve analysis of diagnostic modalities, rather than rating scales based on five to seven ordinal categories or levels of suspicion. Although many investigators have gone over to this method, a discussion of the issues continues. The present work provides a quantitative analysis from the viewpoint of measurement science. MATERIALS AND METHODS: A simple model of the effect of data discretization or quantization on the measurement of the variance of noisy data was developed. Then Monte Carlo simulations of multiple-reader, multiple-case ROC experiments were performed and analyzed in terms of components-of-variance models to investigate the effect of data quantization in that more complex setting. RESULTS: For single-reader studies, discretization into five categories can reduce the precision of ROC measurements by a large amount. The effect may be attenuated in multireader studies. CONCLUSION: More precise measurements of diagnostic detection performance and thus more efficient use of resources are served by good measurement methods. These are promoted by the use of a quasi-continuous rating scale in ROC studies.

Computerized classification of benign and malignant masses on digitized mammograms: a study of robustness.

RATIONALE AND OBJECTIVES: The purpose of this study was to evaluate the robustness of a computerized method developed for the classification of benign and malignant masses with respect to variations in both case mix and film digitization. MATERIALS AND METHODS: The classification method included automated segmentation of mass regions, automated feature-extraction, and automated lesion characterization. The method was evaluated independently with a 110-case database consisting of 50 malignant and 60 benign cases. Mammograms were digitized twice with two different digitizers (Konica and Lumisys). Performance of the method in differentiating benign from malignant masses was evaluated with receiver operating characteristic (ROC) analysis. Effects of variations in both case mix and film digitization on performance of the method also were assessed. RESULTS: Categorization of lesions as malignant or benign with an artificial neural network (or a hybrid) classifier achieved an area under the ROC curve, Az, value of 0.90 (0.94 for the hybrid) on the previous training database in a round-robin evaluation and Az values of 0.82 (0.81) and 0.81 (0.82) on the independent database for the Konica and Lumisys formats, respectively. These differences, however, were not statistically significant (P > .10). CONCLUSION: The computerized method for the classification of lesions on mammograms was robust with respect to variations in case mix and film digitization.

Computer-aided diagnosis of pulmonary nodules: results of a large-scale observer test.

PURPOSE: To determine the effect of computer-aided diagnosis (CAD) on the accuracy of pulmonary nodule detection. MATERIALS AND METHODS: Twenty abnormal chest radiographs, each with a single nodule, and 20 normal radiographs were digitized with a laser scanner. These images were analyzed by using a computer program that indicates areas that may represent pulmonary nodules. The radiographs were displayed on computer workstations in randomized order, and an observer test was performed. One hundred forty-six observers participated, including 23 chest radiologists, 54 other radiologists, 27 radiology residents, and 42 nonradiologists. Cases were interpreted first without and then with the use of CAD. The observers' responses were recorded on a continuous confidence rating scale. Detection accuracy both with and without CAD was evaluated with receiver operating characteristic analysis. RESULTS: The detection accuracy was significantly higher for all categories of observers when CAD was used (chest radiologists, P = 8 x 10(-6); other radiologists, P = 2 x 10(-16); radiology residents, P = 6 x 10(-7); and nonradiologists, P = 8 x 10(-9)). CONCLUSION: CAD has the potential to improve diagnostic accuracy in the detection of lung nodules on digital radiographs.

Effect of dominant features on neural network performance in the classification of mammographic lesions.

Two different classifiers, an artificial neural network (ANN) and a hybrid system (one step rule-based method followed by an artificial neural network) have been investigated to merge computer-extracted features in the task of differentiating between malignant and benign masses. A database consisting of 65 cases (38 malignant and 26 benign) was used in the study. A total of four computer-extracted features--spiculation, margin sharpness and two density-related measures--was used to characterize these masses. Results from our previous study showed that the hybrid system performed better than the ANN classifier. In our current study, to understand the difference between the two classifiers, we investigated their learning and decision-making processes by studying the relationships between the input features and the outputs. A correlation study showed that the outputs from the ANN-alone method correlated strongly with one of the input features (spiculation), yielding a correlation coefficient of 0.91, whereas the correlation coefficients (absolute value) for the other features ranged from 0.19 to 0.40. This strong correlation between the ANN output and spiculation measure indicates that the learning and decision-making processes of the ANN-alone method were dominated by the spiculation measure. Three-dimensional plots of the computer output as functions of the input features demonstrate that the ANN-alone method did not learn as effectively as the hybrid system in differentiating non-spiculated malignant masses from benign masses, thus resulting in an inferior performance at the high sensitivity levels. We found that with a limited database it is detrimental for an ANN to learn the significance of other features in the presence of a dominant feature. The hybrid system, which initially applied a rule concerning the value of the spiculation measure prior to employing an ANN, prevents over-learning from the dominant feature and performed better than the ANN-alone method in merging the computer-extracted features into a correct diagnosis regarding the malignancy of the masses.

Effects of scatter substraction on detection and quantitation in hepatic SPECT.

UNLABELLED: The purpose of this investigation was to examine the effects of subtractive scatter compensation methods on lesion detection and quantitation. METHODS: Receiver operating characteristic (ROC) methodology was used to measure human observer detection accuracy for tumors in the liver using synthetic images. Furthermore, ROC results were compared with mathematical models for detection and activity quantitation to examine (a) the potential for predicting human performance and (b) the relationship between the detection and quantitation tasks. Images with both low and high amounts of scatter were compared with the ideal case of images of primary photons only (i.e., perfect scatter rejection) and with images corrected by subtracting a scatter image estimated by the dual photopeak window method. RESULTS: With low contrast tumors in a low count background, the results showed that scatter subtraction improved quantitation but did not produce statistically significant increases in detection accuracy. However, primary images did produce some statistically significant improvements in detection accuracy when compared with uncorrected images, particularly for high levels of scatter. CONCLUSION: Although scatter subtraction methods may provide improved activity quantitation, they may not significantly improve detection for liver SPECT. The results imply that significant improvement in detection accuracy for the conditions tested may depend on the development of gamma cameras with better scatter rejection.

Improving breast cancer diagnosis with computer-aided diagnosis.

RATIONALE AND OBJECTIVES: The purpose of this study was to test whether computer-aided diagnosis (CAD) can improve radiologists' performance in breast cancer diagnosis. MATERIALS AND METHODS: The computer classification scheme used in this study estimates the likelihood of malignancy for clustered microcalcifications based on eight computer-extracted features obtained from standard-view mammograms. One hundred four histologically verified cases of microcalcifications (46 malignant, 58 benign) in a near-consecutive biopsy series were used in this study. Observer performance was measured on 10 radiologists who read the original standard- and magnification-view mammograms. The computer aid provided a percentage estimate of the likelihood of malignancy. Comparison was made between computer-aided performance and unaided (routine clinical) performance by using receiver operating characteristic (ROC) analysis and by comparing biopsy recommendations. RESULTS: The average ROC curve area (Az) increased from 0.61 without aid to 0.75 with the computer aid (P < .0001). On average, with the computer aid, each observer recommended 6.4 additional biopsies for cases with malignant lesions (P = .0006) and 6.0 fewer biopsies for cases with benign lesions (P = .003). This improvement corresponded to increases in sensitivity (from 73.5% to 87.4%), specificity (from 31.6% to 41.9%), and hypothetical positive biopsy yield (from 46% to 55%). CONCLUSION: CAD can be used to improve radiologists' performance in breast cancer diagnosis.

Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data.

We show that truth-state runs in rank-ordered data constitute a natural categorization of continuously-distributed test results for maximum likelihood (ML) estimation of ROC curves. On this basis, we develop two new algorithms for fitting binormal ROC curves to continuously-distributed data: a true ML algorithm (LABROC4) and a quasi-ML algorithm (LABROC5) that requires substantially less computation with large data sets. Simulation studies indicate that both algorithms produce reliable estimates of the binormal ROC curve parameters a and b, the ROC-area index Az, and the standard errors of those estimates.

Statistical comparison of two ROC-curve estimates obtained from partially-paired datasets.

The authors propose a new generalized method for ROC-curve fitting and statistical testing that allows researchers to utilize all of the data collected in an experimental comparison of two diagnostic modalities, even if some patients have not been studied with both modalities. Their new algorithm, ROCKIT, subsumes previous algorithms as special cases. It conducts all analyses available from previous ROC software and provides 95% confidence intervals for all estimates. ROCKIT was tested on more than half a million computer-simulated datasets of various sizes and configurations representing a range of population ROC curves. The algorithm successfully converged for more than 99.8% of all datasets studied. The type I error rates of the new algorithm's statistical test for differences in Az estimates were excellent for datasets typically encountered in practice, but diverged from alpha for datasets arising from some extreme situations.

Variance-component modeling in the analysis of receiver operating characteristic index estimates.

RATIONALE AND OBJECTIVES: The authors performed this study to clarify and systematize the large number of variances and correlations observable with variance-component models of receiver operating characteristic (ROC) index estimates. MATERIALS AND METHODS: The authors present a variance-component model for ROC index estimates (and for differences between estimates) and show correspondences between the method of experimental replication and the random components in the model. The authors introduce a notation that identifies both the method of replication and, when examining estimate differences, the estimate pairing scheme. RESULTS: For models with three factors (modality, reader, case sample), the authors delineated four methods of replication and eight pairing schemes for generating estimate differences. For each of the resulting 32 replication-pairing combinations, the authors gave expressions for the variance of the difference and for the correlation between the two ROC index estimates. CONCLUSION: The variance-component approach is a useful statistical tool for modeling different sources of variation that contribute to the overall variance of ROC data and index estimates derived from those data.

Improved determination of biplane imaging geometry from two projection images and its application to three-dimensional reconstruction of coronary arterial trees.

A technique has been developed for accurate estimation of three-dimensional (3D) biplane imaging geometry and reconstruction of 3D objects based on two perspective projections acquired at arbitrary orientations, without the need of calibration. The required prior information (i.e., the intrinsic parameters of each single-plane imaging system) for determination of biplane imaging geometry includes (a) the distance between each focal spot and its image plane, SID (the focal-spot to imaging-plane distance); (b) the pixel size, psize (e.g., 0.3 mm/pixel); (c) the distance between the two focal spots ff' or the known 3D distance between two points in the projection images; and (d) for each view, an approximation of the magnification factor, MF (e.g., 1.2), which is the ratio of the SID and the approximate distance of the object to the focal spot. Item (d) is optional but may provide a more accurate estimation if it is available. Given five or more corresponding object points in both views, a constrained nonlinear optimization algorithm is applied to obtain an optimal estimate of the biplane imaging geometry in the form of a rotation matrix R and a translation vector t that characterize the position and orientation of one imaging system relative to the other. With the calculated biplane imaging geometry, 3D spatial information concerning the object can then be reconstructed. The accuracy of this method was evaluated by using a computer-simulated coronary arterial tree and a cube phantom object. Our simulation study showed that a computer-simulated coronary tree can be reconstructed from two views with less than 2 and 8.4 mm root-mean-square (rms) configuration (or relative-position) error and absolute-position error, respectively, even if the input errors in the corresponding 2D points are fairly large (more than two pixels = 0.6 mm). In contrast, input image error of more than one pixel (= 0.3 mm) can yield 3D position errors of 10 cm or more when other existing methods based on linear approaches are employed. For the cube phantom images acquired from a routine biplane system, rms errors in the 3D configuration of the cube and the 3D absolute position were 0.6-0.9 mm and 3.9-5.0 mm, respectively.

The "proper" binormal model: parametric receiver operating characteristic curve estimation with degenerate data.

RATIONALE AND OBJECTIVES: The authors assessed the use of a "proper" binormal model and a new algorithm for maximum-likelihood estimation of receiver operating characteristic (ROC) curves from degenerate data. METHODS: The proper binormal ROC model uses as its decision variable a monotonic transformation of the likelihood ratio that is associated with a pair of normal distributions, thereby ensuring fitted ROC curves with monotonic slope but maintaining a relationship with the conventional binormal model. A computer program entitled PROPROC was used to fit proper ROC curves to data obtained from computer-simulated and real observer studies. RESULTS: ROC indexes such as total area were estimated with PROPROC and compared with the corresponding values obtained from the conventional procedures. CONCLUSION: The proper binormal ROC model overcomes the problem of degeneracy in ROC curve fitting. PROPROC is highly robust and yields ROC estimates with less bias and greater precision than those obtained with the conventional binormal model.

Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation.

RATIONALE AND OBJECTIVES: The authors examined the relationship between the critical P value (alpha) and the empirical type I error rate when using the Dorfman-Berbaum-Metz (DMB) method for analysis of variance in multireader, multimodality receiver operating characteristic (ROC) data. METHODS: The authors developed a linear mixed-effect model to generate continuous, normally distributed random decision variables containing multiple sources (components) of variation. A range of magnitudes for these variance components was used to stimulate experiments in which multiple readers (three or five) read imaged obtained with two modalities from the same set of cases with no re-reading. Three binormal population ROC curves, with areas of 0.962, 0.855, and 0.702, were included. Case-sample sizes ranged from 50 to 400, and either 50% or 10% of cases were actually positive. For each experiment, 2,000 data sets were analyzed by the computer program, and the proportion of 2,000 modality differences that was found to be statistically significant at an alpha level of .05 was tubulated. RESULTS: The test for modality difference performed well for the low and intermediate ROC curves, even with small case samples. For the high ROC curve, the small-sample results were conservative. No relationship between observed type I error rate and the magnitude of data correlation was evident. CONCLUSION: For typical ROC curves, the DBM method is robust in testing for modality effects in the null case, given a sufficient sample size. Instructions for obtaining a free copy of the software are given.

Proper receiver operating characteristic analysis: the bigamma model.

RATIONALE AND OBJECTIVES: The standard binormal model is the most commonly used model for fitting receiver operating characteristic rating data; however, it sometimes produces inappropriate fits that cross the chance line with degenerate data sets. The authors proposed and evaluated a proper constant-shape bigamma model to handle binormal degeneracy. METHODS: Monte Carlo samples were generated from both a standard binormal population model and a proper constant-shape bigamma model in a series of Monte Carlo studies. RESULTS: The results confirm that the standard binormal model is robust in large samples with no degenerate data sets and that the standard binormal model is not robust in small samples because of degenerate data sets. CONCLUSION: A proper constant-shape bigamma model seems to solve the problem of degeneracy without inappropriate chance line crossings. The bigamma fitting model outperformed the standard binormal fitting model in small samples and gave similar results in large samples.

A receiver operating characteristic partial area index for highly sensitive diagnostic tests.

PURPOSE: Area under a receiver operating characteristic (ROC) curve (Az) is widely used as an index of diagnostic performance. However, Az is not a meaningful summary of clinical diagnostic performance when high sensitivity must be maintained clinically. The authors developed a new ROC partial area index, which measures clinical diagnostic performance more meaningfully in such situations, to summarize an ROC curve in only a high-sensitivity region. MATERIALS AND METHODS: The mathematical formation of the partial area index was derived from the conventional binormal model. Statistical tests of apparent differences in this index were formulated analogous to that of Az. One common statistical test involving the partial area index was validated by computer simulations under realistic conditions. RESULTS: An example in mammography illustrates a situation in which the partial area index is more meaningful than Az in measuring clinical diagnostic performance. CONCLUSION: The partial area index can be used as a more meaningful alternative to the conventional Az index for highly sensitive diagnostic tests.

Effect of a computer-aided diagnosis scheme on radiologists' performance in detection of lung nodules on radiographs.

PURPOSE: To evaluate the effect of a computer-aided diagnosis (CAD) scheme on radiologists' performance in the detection of lung nodules, and to examine a new method of receiver operating characteristic (ROC) analysis. MATERIALS AND METHODS: One hundred twenty radiographs (60 normal and 60 abnormal with lung nodules of varying subtlety) were used. Sixteen radiologists (two thoracic, six general, and eight residents) participated in an observer study in which they read both conventional radiographs and digitized radiographs. The radiologists' performance was evaluated with ROC analysis with two different methods (independent testing and sequential testing) and a continuous rating scale. RESULTS: Az (area under the best fit binormal ROC curve when it is plotted in the unit square) values obtained from ROC analysis with and without CAD output were 0.940 and 0.894, respectively, in the independent test and 0.948 and 0.906, respectively, in the sequential test. Findings with both methods indicated that the CAD scheme statistically significantly improved diagnostic accuracy, particularly for radiologists with less experience (P < .001). Reading time was not increased when CAD was used. CONCLUSION: The CAD scheme can assist radiologists in the detection of lung nodules on chest radiographs.

Comparison of receiver operating characteristic curves on the basis of optimal operating points.

RATIONALE AND OBJECTIVES: We developed a method of comparing receiver operating characteristic (ROC) curves on the basis of the utilities associated with their optimal operating points (OOPs). METHODS: OOPs were computed for paired ROC curves on the basis of isocost lines in ROC space with slopes ranging from 0.1 to 3.0. For each pair of OOPs corresponding to a single isocost slope, the difference in costs and the variance of this difference was computed. A sensitivity analysis was thus obtained for the difference between the two curves over a range of isocost slopes. Three published data sets were evaluated using this technique, as well as by comparisons of areas under the curves and of true-positive fractions at fixed false-positive fractions. RESULTS: The OOPs of paired ROC curves often occur at different false-positive fractions. Comparisons of ROC curves on the basis of OOPs may provide results that differ from comparisons of curves at a fixed false-positive fraction. CONCLUSION: ROC curves may be compared on the basis of utilities associated with their OOPs. This comparison of the optimal performance of two diagnostic tests may differ from conventional statistical comparisons.