Continuous versus categorical data for ROC analysis: some quantitative considerations.

RATIONALE AND OBJECTIVES: Several authors have encouraged the use of a quasi-continuous rating scale for data collection in receiver operating characteristic (ROC) curve analysis of diagnostic modalities, rather than rating scales based on five to seven ordinal categories or levels of suspicion. Although many investigators have gone over to this method, a discussion of the issues continues. The present work provides a quantitative analysis from the viewpoint of measurement science. MATERIALS AND METHODS: A simple model of the effect of data discretization or quantization on the measurement of the variance of noisy data was developed. Then Monte Carlo simulations of multiple-reader, multiple-case ROC experiments were performed and analyzed in terms of components-of-variance models to investigate the effect of data quantization in that more complex setting. RESULTS: For single-reader studies, discretization into five categories can reduce the precision of ROC measurements by a large amount. The effect may be attenuated in multireader studies. CONCLUSION: More precise measurements of diagnostic detection performance and thus more efficient use of resources are served by good measurement methods. These are promoted by the use of a quasi-continuous rating scale in ROC studies.

Study design in the evaluation of breast cancer imaging technologies.

RATIONALE AND OBJECTIVES: Bringing a new imaging technology to market is a complex process. Beyond conceptualization and proof of concept, obtaining U.S. Food and Drug Administration (FDA) approval for clinical use depends on the documented experimental establishment of safety and efficacy. In turn, safety and efficacy are evaluated in the context of the intended use of the technology. The purpose of this study was to examine a conceptual framework for technology development and evaluation, focusing on new breast imaging technologies as a highly visible and current case in point. MATERIALS AND METHODS: The FDA views technology development in terms of a preclinical and four clinical phases of assessment. With a concept of research and development as a learning model, this phased-assessment concept of regulatory review against intended use was integrated with a five-level version of a hierarchy-of-efficacy framework for evaluating imaging technologies. Study design and analysis issues are presented in this context, as are approaches to supporting expanded clinical indications and new intended uses after a new technology is marketed. CONCLUSION: Breast imaging technologies may be intended for use as replacements for standard-of-care technologies, as adjuncts, or as complementary technologies. Study designs must be appropriate to establish claims of superiority or equivalence to the standard for the intended use. Screening technologies are ultimately judged on their demonstrated effectiveness in decreasing cause-specific mortality through early detection, but they may be brought to market for other uses on the basis of lesser standards of efficacy (eg, sensitivity, specificity, positive and negative predictive value, and stage of disease detected).

Toward consensus on quantitative assessment of medical imaging systems.

Consensus has been developing over the past few decades on a number of measurements required for the laboratory assessment of medical imaging modalities. Nevertheless, understanding of the connection between these measurements and human observer performance in a broad range of tasks remains far from complete. Focusing primarily on projection radiography to provide concrete examples, this overview indicates areas in which consensus on methodology for physical image-quality measurement has been established. Concepts such as "noise equivalent quanta" (NEQ) and "detective quantum efficiency" (DQE) have been found useful for normalizing physical measurements on an absolute scale and for relating those measurements to the decision performance of a hypothetical "ideal observer" that effectively performs decision tasks from the image data. The connection between ideal observer performance and human performance, as determined by receiver operating characteristic (ROC) analysis, remains to be understood for many clinically relevant tasks.

Quantitative assessment of myocardial ultrasound tissue characterization through receiver operating characteristic analysis of Bayesian classifiers.

OBJECTIVES: This work proposes a self-consistent assessment methodology for quantitative evaluation of any combination of diagnostic features, with the immediate goal of quantitatively assessing the discriminating power in diabetic patients of features derived from ultrasound backscatter from myocardium. BACKGROUND: Four features from analysis of left ventricular myocardial ultrasound backscatter have previously been shown to be sensitive to potentially cardiomyopathic changes in patients with insulin-dependent diabetes mellitus who have no overt heart disease. The measured features were significantly different between such patients and normal control subjects, as well as among groups of such patients with and without systemic complications of the disease. The quantitative discriminating potential of the features was not assessed. METHODS: Multivariate classifier functions were constructed and analyzed by using the methodology of the receiver operating characteristic curve, which allows quantitative assessment of the discriminating power of these features, alone or in combination. The area under the receiver operating characteristic curve--the true positive rate averaged over all false positive rates--was used as a summary measure of performance. RESULTS: In distinguishing patients with insulin-dependent diabetes mellitus from normal control subjects, the most discriminating combination of ultrasound features for the detection of such changes in these patients yielded receiver operating characteristic curves with area measures of approximately 0.80; for such patients with retinopathy the measure increased to 0.90. This performance is comparable to that of many commonly used diagnostic tests. CONCLUSIONS: A self-consistent set of evaluation methodologies has quantitatively demonstrated the sensitivity of four ultrasound backscatter features to otherwise latent changes in myocardial structure that accompany the evolution of insulin-dependent diabetes mellitus. The results are remarkable in themselves and suggest the potential of the features for the general field of cardiac ultrasound tissue characterization.

Objective assessment of image quality. II. Fisher information, Fourier crosstalk, and figures of merit for task performance.

Figures of merit for image quality are derived on the basis of the performance of mathematical observers on specific detection and estimation tasks. The tasks include detection of a known signal superimposed on a known background, detection of a known signal on a random background, estimation of Fourier coefficients of the object, and estimation of the integral of the object over a specified region of interest. The chosen observer for the detection tasks is the ideal linear discriminant, which we call the Hotelling observer. The figures of merit are based on the Fisher information matrix relevant to estimation of the Fourier coefficients and the closely related Fourier crosstalk matrix introduced earlier by Barrett and Gifford [Phys. Med. Biol. 39, 451 (1994)]. A finite submatrix of the infinite Fisher information matrix is used to set Cramer-Rao lower bounds on the variances of the estimates of the first N Fourier coefficients. The figures of merit for detection tasks are shown to be closely related to the concepts of noise-equivalent quanta (NEQ) and generalized NEQ, originally derived for linear, shift-invariant imaging systems and stationary noise. Application of these results to the design of imaging systems is discussed.

Malpractice litigation against child and adolescent psychiatry residency programs, 1981-1991.

A national survey was conducted to investigate malpractice litigation at United States child and adolescent psychiatry residency programs between 1981 and 1991. Fourteen percent of the directors of child and adolescent psychiatry reported at least one malpractice claim during this period. The highest percentage of lawsuits reported was in the northeast. Suicide and sexual abuse of latency age patients by other patients accounted for the most litigation. The mean monetary award was $167,000, and the largest award was $500,000 for discharge of a patient who killed his mother. There has been an increase in malpractice litigation during the past 10 years. Risk management strategies should be implemented to address areas of liability in child and adolescent psychiatry training programs.

SNR and noise measurements for medical imaging: I. A practical approach based on statistical decision theory.

A method of measuring the image quality of medical imaging equipment is considered within the framework of statistical decision theory. In this approach, images are regarded as random vectors and image quality is defined in the context of the image information available for performing a specified detection or discrimination task. The approach provides a means of measuring image quality, as related to the detection of an image detail of interest, without reference to the actual physical mechanisms involved in image formation and without separate measurements of signal transfer characteristics or image noise. The measurement does not, however, consider deterministic errors in the image; they need a separate evaluation for imaging modalities where they are of concern. The detectability of an image detail can be expressed in terms of the ideal observer's signal-to-noise ratio (SNR) at the decision level. Often a good approximation to this SNR can be obtained by employing sub-optimal observers, whose performance correlates well with the performance of human observers as well. In this paper the measurement of SNR is based on implementing algorithmic realizations of specified observers and analysing their responses while actually performing a specified detection task of interest. Three observers are considered: the ideal prewhitening matched filter, the non-prewhitening matched filter, and the DC-suppressing non-prewhitening matched filter. The construction of the ideal observer requires an impractical amount of data and computing, except for the most simple imaging situations. Therefore, the utilization of sub-optimal observers is advised and their performance in detecting a specified signal is discussed. Measurement of noise and SNR has been extended to include temporally varying images and dynamic imaging systems.

Risk classification of life and health insurance applicants with atypical melanocytic hyperplasia or malignant melanoma in situ.

Physicians may attempt to disguise malignant melanoma in situ from insurance companies by diagnosing atypical melanocytic hyperplasia instead. This study indicates that the insurance industry is not misled by atypical melanocytic hyperplasia and treats it equivalently to malignant melanoma in situ. In addition, physicians' failure to diagnose malignant melanoma in situ may result in underestimation of the incidence of malignant melanoma and may cause inadequate initial treatment and patient follow-up.

Photography for the early diagnosis of malignant melanoma in patients with atypical moles.

Several articles have been published that carefully describe techniques for obtaining reliable photographic series of patients with the atypical mole syndrome. Four common methods of total body photography are described. Follow-up studies of the effectiveness of photodocumentation for patients with the atypical mole syndrome are reviewed.

Photographic utilization in dermatology clinics in the United States: a survey of university-based dermatology residency programs.

BACKGROUND: Photography has recently been introduced as an adjunct to the clinical management of patients with the dysplastic nevus syndrome (DNS). OBJECTIVE: Our purpose was to evaluate the methods used and the extent of clinical photography in dermatology residency programs in the United States. METHODS: Nonmilitary accredited dermatology residency programs in the United States were surveyed (73% response rate) regarding utilization and technical aspects of clinical photography. RESULTS: Forty-one percent of respondents used photography for the clinical management of 90% or more of patients with DNS. Twenty-four percent of respondents used photography for the clinical management of all patients with DNS. Eighty-one percent of respondents used slides, and 62% utilized total body photographs. The median number of photographs taken for a patient with DNS was 20. CONCLUSION: Dermatologic photography has been widely adopted for the clinical management of patients with DNS. Failure of the health insurance industry to recognize the value of this procedure may result in its underutilization.

Patient personal injury litigation against dermatology residency programs in the United States, 1964-1988. Implications for future risk-management programs in dermatology and dermatologic surgery.

A national survey reviewing patient injury litigation against US dermatology residency programs revealed that 50% of the respondents had experienced at least one lawsuit between 1964 and 1988. The northeast region reported the most legal activity. Fifty percent of the lawsuits related to therapeutic or surgical complications. Plaintiffs were successful in 37.9% of the lawsuits. The mean award was $26,505, and the largest reported award of $200,000 was for failing to diagnose herpes simplex in an immunocompromised patient. In view of several recent trends in dermatology, the amount of litigation against dermatologists may increase.

Application of information theory to the assessment of computed tomography.

The imaging process has two fundamental stages: detection and display. The detection stage can be quantified rigourously using Shannon's information theory. This requires the contrast scale (CS), modulation transfer function (MTF), and noise power spectrum [N(f)] to be combined into a signal-to-noise ratio (SNR). This results in two fundamental summary figures of merit: the density of noise equivalent quanta (NEQ) in the image and the information bandwidth integral (IBWI). These algorithm-independent measures are used to quantify the recording stage. The display stage is less well understood since it couples to an external observer. Several types of decision makers are treated. Examples are drawn from first and second generation CT, demonstrating that thye are nearly quantum limited for large signals, indicating how their algorithms are matched or mismatched to the geometry, and calculating the contrast-detail diagrams for those decision makers.