Multi-Institutional Evaluation of Digital Tomosynthesis, Dual-Energy Radiography, and Conventional Chest Radiography for the Detection and Management of Pulmonary Nodules.

Purpose To conduct a multi-institutional, multireader study to compare the performance of digital tomosynthesis, dual-energy (DE) imaging, and conventional chest radiography for pulmonary nodule detection and management. Materials and Methods In this binational, institutional review board-approved, HIPAA-compliant prospective study, 158 subjects (43 subjects with normal findings) were enrolled at four institutions. Informed consent was obtained prior to enrollment. Subjects underwent chest computed tomography (CT) and imaging with conventional chest radiography (posteroanterior and lateral), DE imaging, and tomosynthesis with a flat-panel imaging device. Three experienced thoracic radiologists identified true locations of nodules (n = 516, 3-20-mm diameters) with CT and recommended case management by using Fleischner Society guidelines. Five other radiologists marked nodules and indicated case management by using images from conventional chest radiography, conventional chest radiography plus DE imaging, tomosynthesis, and tomosynthesis plus DE imaging. Sensitivity, specificity, and overall accuracy were measured by using the free-response receiver operating characteristic method and the receiver operating characteristic method for nodule detection and case management, respectively. Results were further analyzed according to nodule diameter categories (3-4 mm, >4 mm to 6 mm, >6 mm to 8 mm, and >8 mm to 20 mm). Results Maximum lesion localization fraction was higher for tomosynthesis than for conventional chest radiography in all nodule size categories (3.55-fold for all nodules, P < .001; 95% confidence interval [CI]: 2.96, 4.15). Case-level sensitivity was higher with tomosynthesis than with conventional chest radiography for all nodules (1.49-fold, P < .001; 95% CI: 1.25, 1.73). Case management decisions showed better overall accuracy with tomosynthesis than with conventional chest radiography, as given by the area under the receiver operating characteristic curve (1.23-fold, P < .001; 95% CI: 1.15, 1.32). There were no differences in any specificity measures. DE imaging did not significantly affect nodule detection when paired with either conventional chest radiography or tomosynthesis. Conclusion Tomosynthesis outperformed conventional chest radiography for lung nodule detection and determination of case management; DE imaging did not show significant differences over conventional chest radiography or tomosynthesis alone. These findings indicate performance likely achievable with a range of reader expertise. © RSNA, 2016 Online supplemental material is available for this article.

Breast cancer detection rates using four different types of mammography detectors.

OBJECTIVE: To compare the performance of different types of detectors in breast cancer detection. METHODS: A mammography image set containing subtle malignant non-calcification lesions, biopsy-proven benign lesions, simulated malignant calcification clusters and normals was acquired using amorphous-selenium (a-Se) detectors. The images were adapted to simulate four types of detectors at the same radiation dose: digital radiography (DR) detectors with a-Se and caesium iodide (CsI) convertors, and computed radiography (CR) detectors with a powder phosphor (PIP) and a needle phosphor (NIP). Seven observers marked suspicious and benign lesions. Analysis was undertaken using jackknife alternative free-response receiver operating characteristics weighted figure of merit (FoM). The cancer detection fraction (CDF) was estimated for a representative image set from screening. RESULTS: No significant differences in the FoMs between the DR detectors were measured. For calcification clusters and non-calcification lesions, both CR detectors' FoMs were significantly lower than for DR detectors. The calcification cluster's FoM for CR NIP was significantly better than for CR PIP. The estimated CDFs with CR PIP and CR NIP detectors were up to 15% and 22% lower, respectively, than for DR detectors. CONCLUSION: Cancer detection is affected by detector type, and the use of CR in mammography should be reconsidered. KEY POINTS: The type of mammography detector can affect the cancer detection rates. CR detectors performed worse than DR detectors in mammography. Needle phosphor CR performed better than powder phosphor CR. Calcification clusters detection is more sensitive to detector type than other cancers.

The effect of image processing on the detection of cancers in digital mammography.

OBJECTIVE. The objective of our study was to investigate the effect of image processing on the detection of cancers in digital mammography images. MATERIALS AND METHODS. Two hundred seventy pairs of breast images (both breasts, one view) were collected from eight systems using Hologic amorphous selenium detectors: 80 image pairs showed breasts containing subtle malignant masses; 30 image pairs, biopsy-proven benign lesions; 80 image pairs, simulated calcification clusters; and 80 image pairs, no cancer (normal). The 270 image pairs were processed with three types of image processing: standard (full enhancement), low contrast (intermediate enhancement), and pseudo-film-screen (no enhancement). Seven experienced observers inspected the images, locating and rating regions they suspected to be cancer for likelihood of malignancy. The results were analyzed using a jackknife-alternative free-response receiver operating characteristic (JAFROC) analysis. RESULTS. The detection of calcification clusters was significantly affected by the type of image processing: The JAFROC figure of merit (FOM) decreased from 0.65 with standard image processing to 0.63 with low-contrast image processing (p = 0.04) and from 0.65 with standard image processing to 0.61 with film-screen image processing (p = 0.0005). The detection of noncalcification cancers was not significantly different among the image-processing types investigated (p > 0.40). CONCLUSION. These results suggest that image processing has a significant impact on the detection of calcification clusters in digital mammography. For the three image-processing versions and the system investigated, standard image processing was optimal for the detection of calcification clusters. The effect on cancer detection should be considered when selecting the type of image processing in the future.

Screening mammography: test set data can reasonably describe actual clinical reporting.

PURPOSE: To establish the extent to which test set reading can represent actual clinical reporting in screening mammography. MATERIALS AND METHODS: Institutional ethics approval was granted, and informed consent was obtained from each participating screen reader. The need for informed consent with respect to the use of patient materials was waived. Two hundred mammographic examinations were selected from examinations reported by 10 individual expert screen readers, resulting in 10 reader-specific test sets. Data generated from actual clinical reports were compared with three test set conditions: clinical test set reading with prior images, laboratory test set reading with prior images, and laboratory test set reading without prior images. A further set of five expert screen readers was asked to interpret a common set of images in two identical test set conditions to establish a baseline for intraobserver variability. Confidence scores (from 1 to 4) were assigned to the respective decisions made by readers. Region-of-interest (ROI) figures of merit (FOMs) and side-specific sensitivity and specificity were described for the actual clinical reporting of each reader-specific test set and were compared with those for the three test set conditions. Agreement between pairs of readings was performed by using the Kendall coefficient of concordance. RESULTS: Moderate or acceptable levels of agreement were evident (W = 0.69-0.73, P < .01) when describing group performance between actual clinical reporting and test set conditions that were reasonably close to the established baseline (W = 0.77, P < .01) and were lowest when prior images were excluded. Higher median values for ROI FOMs were demonstrated for the test set conditions than for the actual clinical reporting values; this was possibly linked to changes in sensitivity. CONCLUSION: Reasonable levels of agreement between actual clinical reporting and test set conditions can be achieved, although inflated sensitivity may be evident with test set conditions.

An optimised patient-specific approach to administration of contrast agent for CT pulmonary angiography.

OBJECTIVES: To investigate pulmonary vasculature opacification during CTPA using an optimised patient-specific protocol for administering contrast agent. METHODS: CTPA was performed on 200 patients with suspected PE. Patients were assigned to two protocol groups: protocol A, fixed 80 ml contrast agent; protocol B used a patient-specific approach. The mean cross-sectional opacification profile of 8 central and 11 peripheral pulmonary arteries and veins was measured and the arteriovenous contrast ratio (AVCR) calculated. Protocols were compared using Mann-Whitney U non-parametric statistics. Jack-knife alternative free-response receiver-operating characteristic (JAFROC) analyses assessed diagnostic efficacy. Interobserver variations were investigated using kappa methods. RESULTS: A number of pulmonary arteries demonstrated increases in opacification (P < 0.03) for protocol B compared to A, whilst opacification in the heart and veins was reduced in protocol B (P = 0.05). Increased AVCR in protocol B compared with A was observed at all anatomic locations (P < 0.0002). Increased JAFROC (P < 0.0002) and kappa variation were observed with protocol B (κ = 0.78) compared to A (κ = 0.25). Mean contrast volume was reduced in protocol B (33 ± 9 ml) compared to A (80 ± 1 ml). CONCLUSIONS: Significant improvements in visualisation of the pulmonary vasculature can be achieved with a low volume of contrast agent using injection timing based on a patient-specific contrast formula. KEY POINTS: • Optimal opacification of the pulmonary arteries is essential for CT pulmonary angiography. • Matching timing with vessel dynamics significantly improves vessel opacification. • This leads to increased arterial opacification and reduced venous opacification. • This can also lead to a reduced volume of contrast agent.

Scalable, modular continuous wave functional near-infrared spectroscopy system (Spotlight).

Significance: We present a fiberless, portable, and modular continuous wave-functional near-infrared spectroscopy system, Spotlight, consisting of multiple palm-sized modules-each containing high-density light-emitting diode and silicon photomultiplier detector arrays embedded in a flexible membrane that facilitates optode coupling to scalp curvature. Aim: Spotlight's goal is to be a more portable, accessible, and powerful functional near-infrared spectroscopy (fNIRS) device for neuroscience and brain-computer interface (BCI) applications. We hope that the Spotlight designs we share here can spur more advances in fNIRS technology and better enable future non-invasive neuroscience and BCI research. Approach: We report sensor characteristics in system validation on phantoms and motor cortical hemodynamic responses in a human finger-tapping experiment, where subjects wore custom 3D-printed caps with two sensor modules. Results: The task conditions can be decoded offline with a median accuracy of 69.6%, reaching 94.7% for the best subject, and at a comparable accuracy in real time for a subset of subjects. We quantified how well the custom caps fitted to each subject and observed that better fit leads to more observed task-dependent hemodynamic response and better decoding accuracy. Conclusions: The advances presented here should serve to make fNIRS more accessible for BCI applications.

Effect of image quality on calcification detection in digital mammography.

PURPOSE: This study aims to investigate if microcalcification detection varies significantly when mammographic images are acquired using different image qualities, including: different detectors, dose levels, and different image processing algorithms. An additional aim was to determine how the standard European method of measuring image quality using threshold gold thickness measured with a CDMAM phantom and the associated limits in current EU guidelines relate to calcification detection. METHODS: One hundred and sixty two normal breast images were acquired on an amorphous selenium direct digital (DR) system. Microcalcification clusters extracted from magnified images of slices of mastectomies were electronically inserted into half of the images. The calcification clusters had a subtle appearance. All images were adjusted using a validated mathematical method to simulate the appearance of images from a computed radiography (CR) imaging system at the same dose, from both systems at half this dose, and from the DR system at quarter this dose. The original 162 images were processed with both Hologic and Agfa (Musica-2) image processing. All other image qualities were processed with Agfa (Musica-2) image processing only. Seven experienced observers marked and rated any identified suspicious regions. Free response operating characteristic (FROC) and ROC analyses were performed on the data. The lesion sensitivity at a nonlesion localization fraction (NLF) of 0.1 was also calculated. Images of the CDMAM mammographic test phantom were acquired using the automatic setting on the DR system. These images were modified to the additional image qualities used in the observer study. The images were analyzed using automated software. In order to assess the relationship between threshold gold thickness and calcification detection a power law was fitted to the data. RESULTS: There was a significant reduction in calcification detection using CR compared with DR: the alternative FROC (AFROC) area decreased from 0.84 to 0.63 and the ROC area decreased from 0.91 to 0.79 (p < 0.0001). This corresponded to a 30% drop in lesion sensitivity at a NLF equal to 0.1. Detection was also sensitive to the dose used. There was no significant difference in detection between the two image processing algorithms used (p > 0.05). It was additionally found that lower threshold gold thickness from CDMAM analysis implied better cluster detection. The measured threshold gold thickness passed the acceptable limit set in the EU standards for all image qualities except half dose CR. However, calcification detection varied significantly between image qualities. This suggests that the current EU guidelines may need revising. CONCLUSIONS: Microcalcification detection was found to be sensitive to detector and dose used. Standard measurements of image quality were a good predictor of microcalcification cluster detection.

Diagnostic efficacy of handheld devices for emergency radiologic consultation.

OBJECTIVE: Orthopedic injury and intracranial hemorrhage are commonly encountered in emergency radiology, and accurate and timely diagnosis is important. The purpose of this study was to determine whether the diagnostic accuracy of handheld computing devices is comparable to that of monitors that might be used in emergency teleconsultation. SUBJECTS AND METHODS: Two handheld devices, a Dell Axim personal digital assistant (PDA) and an Apple iPod Touch device, were studied. The diagnostic efficacy of each device was tested against that of secondary-class monitors (primary class being clinical workstation display) for each of two image types-posteroanterior wrist radiographs and slices from CT of the brain-yielding four separate observer performance studies. Participants read a bank of 30 wrist or brain images searching for a specific abnormality (distal radial fracture, fresh intracranial bleed) and rated their confidence in their decisions. A total of 168 readings by examining radiologists of the American Board of Radiology were gathered, and the results were subjected to receiver operating characteristics analysis. RESULTS: In the PDA brain CT study, the scores of PDA readings were significantly higher than those of monitor readings for all observers (p < or = 0.01) and for radiologists who were not neuroradiology specialists (p < or = 0.05). No statistically significant differences between handheld device and monitor findings were found for the PDA wrist images or in the iPod Touch device studies, although some comparisons approached significance. CONCLUSION: Handheld devices show promise in the field of emergency teleconsultation for detection of basic orthopedic injuries and intracranial hemorrhage. Further investigation is warranted.

Detection of urinary stones at reduced radiation exposure: a phantom study comparing computed radiography and a low-dose digital radiography linear slit scanning system.

OBJECTIVE: In this experimental study we assessed the diagnostic performance of digital linear slit scanning radiography compared with computed radiography (CR) for the detection of urinary calculi in an anthropomorphic phantom imitating patients weighing approximately 58-88 kg. CONCLUSION: Compared with CR, linear slit scanning radiography is superior for the detection of urinary stones and may be used for pretreatment localization and follow-up at a lower patient exposure.

An improved method for simulating microcalcifications in digital mammograms.

The assessment of the performance of a digital mammography system requires an observer study with a relatively large number of cases with known truth which is often difficult to assemble. Several investigators have developed methods for generating hybrid abnormal images containing simulated microcalcifications. This article addresses some of the limitations of earlier methods. The new method is based on digital images of needle biopsy specimens. Since the specimens are imaged separately from the breast, the microcalcification attenuation profile scan is deduced without the effects of over and underlying tissues. The resulting templates are normalized for image acquisition specific parameters and reprocessed to simulate microcalcifications appropriate to other imaging systems, with different x-ray, detector and image processing parameters than the original acquisition system. This capability is not shared by previous simulation methods that have relied on extracting microcalcifications from breast images. The method was validated by five experienced mammographers who compared 59 pairs of simulated and real microcalcifications in a two-alternative forced choice task designed to test if they could distinguish the real from the simulated lesions. They also classified the shapes of the microcalcifications according to a standardized clinical lexicon. The observed probability of correct choice was 0.415, 95% confidence interval (0.284, 0.546), showing that the radiologists were unable to distinguish the lesions. The shape classification revealed substantial agreement with the truth (mean kappa = 0.70), showing that we were able to accurately simulate the lesion morphology. While currently limited to single microcalcifications, the method is extensible to more complex clusters of microcalcifications and to three-dimensional images. It can be used to objectively assess an imaging technology, especially with respect to its ability to adequately visualize the morphology of the lesions, which is a critical factor in the benign versus malignant classification of a lesion detected in screening mammography.

The impact of acoustic noise found within clinical departments on radiology performance.

RATIONALE AND OBJECTIVES: In recent years, there has been increasing interest in the impact of environmental factors such as ambient light on radiologist performance. One commonly encountered distractor found within all clinical departments that has received little or no attention is acoustic noise. MATERIALS AND METHODS: The present work records the level of noises encountered within environments where radiologic images are viewed and establishes the impact of a clinically relevant level of noise on the ability of radiologists to perform a typical diagnostic task. Noise levels were recorded 10 times within each of 14 environments, 11 of which were locations where radiologic images are judged. Thirty chest images were then presented to 26 senior radiologists, who were asked to detect up to three nodular lesions within 30 posteroanterior chest x-ray images in the absence and presence of noise at an amplitude demonstrated in the clinical environment. Jackknife free-response receiver-operating characteristic analyses was performed on the free-response data. RESULTS: The results demonstrated that noise amplitudes rarely exceeded that encountered with normal conversation with the maximum mean value for an image-viewing environment being 56.1 dB. This level of noise had no impact on the ability of radiologists to identify chest lesions with figure of merits of 0.68, 0.69, and 0.68 with noise and 0.65, 0.68, and 0.67 without noise for chest radiologists, nonchest radiologists, and all radiologists, respectively. Equally, no differences were seen for false-positive and false-negative scores or on the time required to judge the images. CONCLUSION: These findings suggest that noise at levels encountered within areas where radiologic images are viewed is not a major distractor within the reporting environment, but the need for further work has been identified.

Evaluating computer-aided detection algorithms.

Computer-aided detection (CAD) has been attracting extensive research interest during the last two decades. It is recognized that the full potential of CAD can only be realized by improving the performance and robustness of CAD algorithms and this requires good evaluation methodology that would permit CAD designers to optimize their algorithms. Free-response receiver operating characteristic (FROC) curves are widely used to assess CAD performance, however, evaluation rarely proceeds beyond determination of lesion localization fraction (sensitivity) at an arbitrarily selected value of nonlesion localizations (false marks) per image. This work describes a FROC curve fitting procedure that uses a recent model of visual search that serves as a framework for the free-response task. A maximum likelihood procedure for estimating the parameters of the model from free-response data and fitting CAD generated FROC curves was implemented. Procedures were implemented to estimate two figures of merit and associated statistics such as 95% confidence intervals and goodness of fit. One of the figures of merit does not require the arbitrary specification of an operating point at which to evaluate CAD performance. For comparison a related method termed initial detection and candidate analysis was also implemented that is applicable when all suspicious regions are reported. The two methods were tested on seven mammography CAD data sets and both yielded good to excellent fits. The search model approach has the advantage that it can potentially be applied to radiologist generated free-response data where not all suspicious regions are reported, only the ones that are deemed sufficiently suspicious to warrant clinical follow-up. This work represents the first practical application of the search model to an important evaluation problem in diagnostic radiology. Software based on this work is expected to benefit CAD developers working in diverse areas of medical imaging.

Spatial localization accuracy of radiologists in free-response studies: Inferring perceptual FROC curves from mark-rating data.

RATIONALE AND OBJECTIVES: Free-response data consist of a set of mark-ratings pairs. Before analysis, the data are classified or "scored" into lesion and non-lesion localizations. The scoring is done by choosing an acceptance-radius and classifying marks within the acceptance-radius of lesion centers as lesion localizations, and all other marks are classified as non-lesion localizations. The scored data are plotted as a free-response receiver operating characteristic (FROC) curve, essentially a plot of appropriately normalized numbers of lesion localizations vs. non-lesion localizations. Scored FROC curves are frequently used to compare imaging systems and computer-aided detection (CAD) algorithms. However, the choice of acceptance-radius is arbitrary. This makes it difficult to compare curves from different studies and to estimate true performance. MATERIALS AND METHODS: To resolve this issue the concept of two types of marks is introduced: perceptual hits and perceptual misses. A perceptual hit is a mark made in response to the observer seeing the lesion. A perceptual miss is a mark made in response to the observer seeing a (lesion-like) non-lesion. A method of estimating the most probable numbers of perceptual hits and misses is described. This allows one to plot a perceptual FROC operating point and by extension a perceptual FROC curve. Unlike a scored FROC operating point, a perceptual point is independent of the choice of acceptance-radius. The method does not allow one to identify individual marks as perceptual hits or misses-only the most probable numbers. It is based on a three-parameter statistical model of the spatial distributions of perceptual hits and misses relative to lesion centers. RESULTS: The method has been applied to an observer dataset in which mammographers and residents with different levels of experience were asked to locate lesions in mammograms. The perceptual operating points suggest superior performance for the mammographers and equivalent performance for residents in the first and second mammography rotations. These results and the model validation are preliminary as they are based on a small dataset. CONCLUSION: The significance of this study is showing that it is possible to probabilistically determine if a mark resulted from seeing a lesion or a non-lesion. Using the method developed in this study one could perform acceptance-radius independent estimation of observer performance.

A bivariate contaminated binormal model for robust fitting of proper ROC curves to a pair of correlated, possibly degenerate, ROC datasets.

PURPOSE: The objective was to design and implement a bivariate extension to the contaminated binormal model (CBM) to fit paired receiver operating characteristic (ROC) datasets-possibly degenerate-with proper ROC curves. Paired datasets yield two correlated ratings per case. Degenerate datasets have no interior operating points and proper ROC curves do not inappropriately cross the chance diagonal. The existing method, developed more than three decades ago utilizes a bivariate extension to the binormal model, implemented in CORROC2 software, which yields improper ROC curves and cannot fit degenerate datasets. CBM can fit proper ROC curves to unpaired (i.e., yielding one rating per case) and degenerate datasets, and there is a clear scientific need to extend it to handle paired datasets. METHODS: In CBM, nondiseased cases are modeled by a probability density function (pdf) consisting of a unit variance peak centered at zero. Diseased cases are modeled with a mixture distribution whose pdf consists of two unit variance peaks, one centered at positive µ with integrated probability α, the mixing fraction parameter, corresponding to the fraction of diseased cases where the disease was visible to the radiologist, and one centered at zero, with integrated probability (1-α), corresponding to disease that was not visible. It is shown that: (a) for nondiseased cases the bivariate extension is a unit variances bivariate normal distribution centered at (0,0) with a specified correlation ρ1 ; (b) for diseased cases the bivariate extension is a mixture distribution with four peaks, corresponding to disease not visible in either condition, disease visible in only one condition, contributing two peaks, and disease visible in both conditions. An expression for the likelihood function is derived. A maximum likelihood estimation (MLE) algorithm, CORCBM, was implemented in the R programming language that yields parameter estimates and the covariance matrix of the parameters, and other statistics. A limited simulation validation of the method was performed. RESULTS: CORCBM and CORROC2 were applied to two datasets containing nine readers each contributing paired interpretations. CORCBM successfully fitted the data for all readers, whereas CORROC2 failed to fit a degenerate dataset. All fits were visually reasonable. All CORCBM fits were proper, whereas all CORROC2 fits were improper. CORCBM and CORROC2 were in agreement (a) in declaring only one of the nine readers as having significantly different performances in the two modalities; (b) in estimating higher correlations for diseased cases than for nondiseased ones; and (c) in finding that the intermodality correlation estimates for nondiseased cases were consistent between the two methods. All CORCBM fits yielded higher area under curve (AUC) than the CORROC2 fits, consistent with the fact that a proper ROC model like CORCBM is based on a likelihood-ratio-equivalent decision variable, and consequently yields higher performance than the binormal model-based CORROC2. The method gave satisfactory fits to four simulated datasets. CONCLUSIONS: CORCBM is a robust method for fitting paired ROC datasets, always yielding proper ROC curves, and able to fit degenerate datasets.

Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method.

RATIONALE AND OBJECTIVES: The free-response paradigm is being increasingly used in the assessment of medical imaging systems. The currently implemented method of analyzing the data, namely jackknife free-response (JAFROC) analysis, has some validation and applicability limitations. The purpose of this work is to address these limitations. MATERIALS AND METHODS: The general principles of modality evaluation and methodology validation are reviewed. A model for simulating free-response data was used to test the statistical validity of several methods of analyzing the data. The methods differed only in the choice of the figure of merit used to quantify performance. Statistical validity was judged by investigating the behaviors of the methods under null hypothesis conditions of no difference between modalities. RESULTS: The validity of the different methods of analyzing the data was found to be dependent on the choice of figure of merit. A figure of merit is identified that accommodates abnormal images with multiple (one or more) lesions, detections of which could have different clinical significances (weights). This figure of merit is shown to be statistically valid. An extension of the analysis to single reader interpretations of images from different modalities is also shown to be statistically valid. CONCLUSION: With the validated enhancements, JAFROC is expected to be of greater utility to users of the free-response method. The extension to single-reader interpretations should be of particular value to developers of image processing algorithms, including developers of computer-aided diagnosis algorithms.

An alternate method for using a visual discrimination model (VDM) to optimize soft-copy display image quality.

Researchers have developed visual discrimination models (VDMs) that can predict a human observer's ability to detect a target object superposed on an image. These models incorporate sophisticated knowledge of the properties of the human visual system. In the predictive approach, termed conventional VDM usage, two input images with and without a target are analyzed by an algorithm that calculates a just-noticeable-difference (JND) index, which is a taken as a measure of the detectability of the target. A new method of using the VDM is described, termed channelized VDM, which involves finding the linear combination of the VDM-generated channels (which are not used in conventional VDM analysis) that has optimal classification ability between normal and abnormal images. The classification ability can be measured using receiver operating characteristic (ROC) or two alternative forced choice (2AFC) experiments, and in special cases they can also be predicted by signal detection theory (SDT) based model-observer methods. In this study simulated background and nodule containing regions were used to validate the new method. It was found that the channelized VDM predictions were in excellent qualitative agreement with human-observer validated SDT predictions. Either VDM method (conventional or channelized) has potential applicability to soft-copy display optimization. An advantage of any VDM-based approach is that complex effects, such as visual masking, are automatically accounted for, which effects are usually not included in SDT-based methods.

On the meaning of the weighted alternative free-response operating characteristic figure of merit.

PURPOSE: The free-response receiver operating characteristic (FROC) method is being increasingly used to evaluate observer performance in search tasks. Data analysis requires definition of a figure of merit (FOM) quantifying performance. While a number of FOMs have been proposed, the recommended one, namely, the weighted alternative FROC (wAFROC) FOM, is not well understood. The aim of this work is to clarify the meaning of this FOM by relating it to the empirical area under a proposed wAFROC curve. METHODS: The weighted wAFROC FOM is defined in terms of a quasi-Wilcoxon statistic that involves weights, coding the clinical importance, assigned to each lesion. A new wAFROC curve is proposed, the y-axis of which incorporates the weights, giving more credit for marking clinically important lesions, while the x-axis is identical to that of the AFROC curve. An expression is derived relating the area under the empirical wAFROC curve to the wAFROC FOM. Examples are presented with small numbers of cases showing how AFROC and wAFROC curves are affected by correct and incorrect decisions and how the corresponding FOMs credit or penalize these decisions. The wAFROC, AFROC, and inferred ROC FOMs were applied to three clinical data sets involving multiple reader FROC interpretations in different modalities. RESULTS: It is shown analytically that the area under the empirical wAFROC curve equals the wAFROC FOM. This theorem is the FROC analog of a well-known theorem developed in 1975 for ROC analysis, which gave meaning to a Wilcoxon statistic based ROC FOM. A similar equivalence applies between the area under the empirical AFROC curve and the AFROC FOM. The examples show explicitly that the wAFROC FOM gives equal importance to all diseased cases, regardless of the number of lesions, a desirable statistical property not shared by the AFROC FOM. Applications to the clinical data sets show that the wAFROC FOM yields results comparable to that using the AFROC FOM. CONCLUSIONS: The equivalence theorem gives meaning to the weighted AFROC FOM, namely, it is identical to the empirical area under weighted AFROC curve.

The relationship between cancer detection in mammography and image quality measurements.

PURPOSE: To investigate the relationship between image quality measurements and the clinical performance of digital mammographic systems. METHODS: Mammograms containing subtle malignant non-calcification lesions and simulated malignant calcification clusters were adapted to appear as if acquired by four types of detector. Observers searched for suspicious lesions and gave these a malignancy score. Analysis was undertaken using jackknife alternative free-response receiver operating characteristics weighted figure of merit (FoM). Images of a CDMAM contrast-detail phantom were adapted to appear as if acquired using the same four detectors as the clinical images. The resultant threshold gold thicknesses were compared to the FoMs using a linear regression model and an F-test was used to find if the gradient of the relationship was significantly non-zero. RESULTS: The detectors with the best image quality measurement also had the highest FoM values. The gradient of the inverse relationship between FoMs and threshold gold thickness for the 0.25mm diameter disk was significantly different from zero for calcification clusters (p=0.027), but not for non-calcification lesions (p=0.11). Systems performing just above the minimum image quality level set in the European Guidelines for Quality Assurance in Breast Cancer Screening and Diagnosis resulted in reduced cancer detection rates compared to systems performing at the achievable level. CONCLUSIONS: The clinical effectiveness of mammography for the task of detecting calcification clusters was found to be linked to image quality assessment using the CDMAM phantom. The European Guidelines should be reviewed as the current minimum image quality standards may be too low.

A comparison of two data analyses from two observer performance studies using Jackknife ROC and JAFROC.

The authors compared two methodological approaches, Jackknife ROC and JAFROC, in analyzing data ascertained during FROC (free-response receiver operating characteristics) type studies. Observer rating data obtained from two observer performance studies were analyzed. During the first study, seven radiologists interpreted 120 mammography examinations depicting 57 masses under five different conditions with and without the results of computer-aided detection (CAD). In the second study, eight radiologists interpreted 110 examinations depicting 51 masses under six different display conditions with and without CAD results. Readers rated the detection task in a FROC type response. Jackknife ROC (using the software of LABMRMC with the highest rating per case) and JAFROC were used to compute differences, if any, in summary performance levels among all reading modes in each study as well as for all paired data sets. The results of the different analytical approaches are compared. The overall results for all modes were significantly different for the first study (p < 0.05) and not significant (p > 0.05) for the second study using either analytical approach. In the first study, the performance levels represented by three paired data sets were significantly different (p < 0.05) when computed using LABMRMC and four pairs were significantly different (p < 0.05) using JAFROC. In eight of ten pairs, JAFROC produced lower p values than LABMRMC. In the second study, LABMRMC showed no significant differences for any paired data sets and JAFROC showed a significant difference for one pair. In 15 of 16 pairs, p values computed by JAFROC were lower than those computed by LABMRMC.

Assessment of display performance for medical imaging systems: executive summary of AAPM TG18 report.

Digital imaging provides an effective means to electronically acquire, archive, distribute, and view medical images. Medical imaging display stations are an integral part of these operations. Therefore, it is vitally important to assure that electronic display devices do not compromise image quality and ultimately patient care. The AAPM Task Group 18 (TG18) recently published guidelines and acceptance criteria for acceptance testing and quality control of medical display devices. This paper is an executive summary of the TG18 report. TG18 guidelines include visual, quantitative, and advanced testing methodologies for primary and secondary class display devices. The characteristics, tested in conjunction with specially designed test patterns (i.e., TG18 patterns), include reflection, geometric distortion, luminance, the spatial and angular dependencies of luminance, resolution, noise, glare, chromaticity, and display artifacts. Geometric distortions are evaluated by linear measurements of the TG18-QC test pattern, which should render distortion coefficients less than 2%/5% for primary/secondary displays, respectively. Reflection measurements include specular and diffuse reflection coefficients from which the maximum allowable ambient lighting is determined such that contrast degradation due to display reflection remains below a 20% limit and the level of ambient luminance (Lamb) does not unduly compromise luminance ratio (LR) and contrast at low luminance levels. Luminance evaluation relies on visual assessment of low contrast features in the TG18-CT and TG18-MP test patterns, or quantitative measurements at 18 distinct luminance levels of the TG18-LN test patterns. The major acceptable criteria for primary/ secondary displays are maximum luminance of greater than 170/100 cd/m2, LR of greater than 250/100, and contrast conformance to that of the grayscale standard display function (GSDF) of better than 10%/20%, respectively. The angular response is tested to ascertain the viewing cone within which contrast conformance to the GSDF is better than 30%/60% and LR is greater than 175/70 for primary/secondary displays, or alternatively, within which the on-axis contrast thresholds of the TG18-CT test pattern remain discernible. The evaluation of luminance spatial uniformity at two distinct luminance levels across the display faceplate using TG18-UNL test patterns should yield nonuniformity coefficients smaller than 30%. The resolution evaluation includes the visual scoring of the CX test target in the TG18-QC or TG18-CX test patterns, which should yield scores greater than 4/6 for primary/secondary displays. Noise evaluation includes visual evaluation of the contrast threshold in the TG18-AFC test pattern, which should yield a minimum of 3/2 targets visible for primary/secondary displays. The guidelines also include methodologies for more quantitative resolution and noise measurements based on MTF and NPS analyses. The display glare test, based on the visibility of the low-contrast targets of the TG18-GV test pattern or the measurement of the glare ratio (GR), is expected to yield scores greater than 3/1 and GRs greater than 400/150 for primary/secondary displays. Chromaticity, measured across a display faceplate or between two display devices, is expected to render a u',v' color separation of less than 0.01 for primary displays. The report offers further descriptions of prior standardization efforts, current display technologies, testing prerequisites, streamlined procedures and timelines, and TG18 test patterns.