Improving Accuracy and Efficiency with Concurrent Use of Artificial Intelligence for Digital Breast Tomosynthesis.

PURPOSE: To evaluate the use of artificial intelligence (AI) to shorten digital breast tomosynthesis (DBT) reading time while maintaining or improving accuracy. MATERIALS AND METHODS: A deep learning AI system was developed to identify suspicious soft-tissue and calcified lesions in DBT images. A reader study compared the performance of 24 radiologists (13 of whom were breast subspecialists) reading 260 DBT examinations (including 65 cancer cases) both with and without AI. Readings occurred in two sessions separated by at least 4 weeks. Area under the receiver operating characteristic curve (AUC), reading time, sensitivity, specificity, and recall rate were evaluated with statistical methods for multireader, multicase studies. RESULTS: Radiologist performance for the detection of malignant lesions, measured by mean AUC, increased 0.057 with the use of AI (95% confidence interval [CI]: 0.028, 0.087; P < .01), from 0.795 without AI to 0.852 with AI. Reading time decreased 52.7% (95% CI: 41.8%, 61.5%; P < .01), from 64.1 seconds without to 30.4 seconds with AI. Sensitivity increased from 77.0% without AI to 85.0% with AI (8.0%; 95% CI: 2.6%, 13.4%; P < .01), specificity increased from 62.7% without to 69.6% with AI (6.9%; 95% CI: 3.0%, 10.8%; noninferiority P < .01), and recall rate for noncancers decreased from 38.0% without to 30.9% with AI (7.2%; 95% CI: 3.1%, 11.2%; noninferiority P < .01). CONCLUSION: The concurrent use of an accurate DBT AI system was found to improve cancer detection efficacy in a reader study that demonstrated increases in AUC, sensitivity, and specificity and a reduction in recall rate and reading time.© RSNA, 2019See also the commentary by Hsu and Hoyt in this issue.

Concurrent Computer-Aided Detection Improves Reading Time of Digital Breast Tomosynthesis and Maintains Interpretation Performance in a Multireader Multicase Study.

OBJECTIVE: Digital breast tomosynthesis (DBT) is more accurate than full-field digital mammography alone but requires a longer reading time. A radiologist reader study evaluated the use of concurrent computer-aided detection (CAD) to shorten the reading time while maintaining interpretation performance. MATERIALS AND METHODS: A CAD system was developed to detect suspicious soft-tissue densities in DBT planes. Abnormalities are extracted from the plane in which they are detected and blended into the corresponding synthetic image. The study used an enriched sample of 240 DBT cases with 68 malignancies in 61 patients. Twenty radiologists retrospectively reviewed all 240 cases in a multireader multicase crossover design to compare reading time and performance with and without CAD. The performance of CAD alone was also evaluated. RESULTS: Reading time improved by 29.2% with CAD (95% CI, 21.1-36.5%; p < 0.01). Reader performance, measured by ROC AUC, was noninferior with CAD (p < 0.01). The mean AUC increased from 0.841 without to 0.850 with CAD (95% CI, -0.012 to 0.030). Mean sensitivity increased from 0.847 without to 0.871 with CAD (difference 95% CI, -0.005 to 0.055), showing a 0.033 increase in sensitivity for cases with soft-tissue densities (95% CI, -0.002 to 0.068). Mean specificity decreased from 0.527 without to 0.509 with CAD (difference 95% CI, -0.041 to 0.005), and mean recall rate for noncancers slightly increased from 0.474 without to 0.492 with CAD (difference 95% CI, -0.006 to 0.041). CONCLUSION: Concurrent use of CAD with DBT resulted in 29.2% faster reading time, while maintaining reader interpretation performance.

Improving digital breast tomosynthesis reading time: A pilot multi-reader, multi-case study using concurrent Computer-Aided Detection (CAD).

PURPOSE: Evaluate concurrent Computer-Aided Detection (CAD) with Digital Breast Tomosynthesis (DBT) to determine impact on radiologist performance and reading time. MATERIALS AND METHODS: The CAD system detects and extracts suspicious masses, architectural distortions and asymmetries from DBT planes that are blended into corresponding synthetic images to form CAD-enhanced synthetic images. Review of CAD-enhanced images and navigation to corresponding planes to confirm or dismiss potential lesions allows radiologists to more quickly review DBT planes. A retrospective, crossover study with and without CAD was conducted with six radiologists who read an enriched sample of 80 DBT cases including 23 malignant lesions in 21 women. Area Under the Receiver Operating Characteristic (ROC) Curve (AUC) compared the readings with and without CAD to determine the effect of CAD on overall interpretation performance. Sensitivity, specificity, recall rate and reading time were also assessed. Multi-reader, multi-case (MRMC) methods accounting for correlation and requiring correct lesion localization were used to analyze all endpoints. AUCs were based on a 0-100% probability of malignancy (POM) score. Sensitivity and specificity were based on BI-RADS scores, where 3 or higher was positive. RESULTS: Average AUC across readers without CAD was 0.854 (range: 0.785-0.891, 95% confidence interval (CI): 0.769,0.939) and 0.850 (range: 0.746-0.905, 95% CI: 0.751,0.949) with CAD (95% CI for difference: -0.046,0.039), demonstrating non-inferiority of AUC. Average reduction in reading time with CAD was 23.5% (95% CI: 7.0-37.0% improvement), from an average 48.2 (95% CI: 39.1,59.6) seconds without CAD to 39.1 (95% CI: 26.2,54.5) seconds with CAD. Per-patient sensitivity was the same with and without CAD (0.865; 95% CI for difference: -0.070,0.070), and there was a small 0.022 improvement (95% CI for difference: -0.046,0.089) in per-lesion sensitivity from 0.790 without CAD to 0.812 with CAD. A slight reduction in specificity with a -0.014 difference (95% CI for difference: -0.079,0.050) and a small 0.025 increase (95% CI for difference: -0.036,0.087) in recall rate in non-cancer cases were observed with CAD. CONCLUSIONS: Concurrent CAD resulted in faster reading time with non-inferiority of radiologist interpretation performance. Radiologist sensitivity, specificity and recall rate were similar with and without CAD.

Estimation of Observer Performance for Reduced Radiation Dose Levels in CT: Eliminating Reduced Dose Levels That Are Too Low Is the First Step.

RATIONALE AND OBJECTIVES: This study aims to estimate observer performance for a range of dose levels for common computed tomography (CT) examinations (detection of liver metastases or pulmonary nodules, and cause of neurologic deficit) to prioritize noninferior dose levels for further analysis. MATERIALS AND METHODS: Using CT data from 131 examinations (abdominal CT, 44; chest CT, 44; head CT, 43), CT images corresponding to 4%-100% of the routine clinical dose were reconstructed with filtered back projection or iterative reconstruction. Radiologists evaluated CT images, marking specified targets, providing confidence scores, and grading image quality. Noninferiority was assessed using reference standards, reader agreement rules, and jackknife alternative free-response receiver operating characteristic figures of merit. Reader agreement required that a majority of readers at lower dose identify target lesions seen by the majority of readers at routine dose. RESULTS: Reader agreement identified dose levels lower than 50% and 4% to have inadequate performance for detection of hepatic metastases and pulmonary nodules, respectively, but could not exclude any low dose levels for head CT. Estimated differences in jackknife alternative free-response receiver operating characteristic figures of merit between routine and lower dose configurations found that only the lowest dose configurations tested (ie, 30%, 4%, and 10% of routine dose levels for abdominal, chest, and head CT examinations, respectively) did not meet criteria for noninferiority. At lower doses, subjective image quality declined before observer performance. Iterative reconstruction was only beneficial when filtered back projection did not result in noninferior performance. CONCLUSION: Opportunity exists for substantial radiation dose reduction using existing CT technology for common diagnostic tasks.

Journal club: molecular breast imaging at reduced radiation dose for supplemental screening in mammographically dense breasts.

OBJECTIVE. The purpose of this study was to assess the diagnostic performance of supplemental screening molecular breast imaging (MBI) in women with mammographically dense breasts after system modifications to permit radiation dose reduction. SUBJECTS AND METHODS. A total of 1651 asymptomatic women with mammographically dense breasts on prior mammography underwent screening mammography and adjunct MBI performed with 300-MBq (99m)Tc-sestamibi and a direct-conversion (cadmium zinc telluride) gamma camera, both interpreted independently. The cancer detection rate, sensitivity, specificity, and positive predictive value of biopsies performed (PPV3) were determined. RESULTS. In 1585 participants with a complete reference standard, 21 were diagnosed with cancer: two detected by mammography only, 14 by MBI only, three by both modalities, and two by neither. Of 14 participants with cancers detected only by MBI, 11 had invasive disease (median size, 0.9 cm; range, 0.5-4.1 cm). Nine of 11 (82%) were node negative, and two had bilateral cancers. With the addition of MBI to mammography, the overall cancer detection rate (per 1000 screened) increased from 3.2 to 12.0 (p < 0.001) (supplemental yield 8.8). The invasive cancer detection rate increased from 1.9 to 8.8 (p < 0.001) (supplemental yield 6.9), a relative increase of 363%, while the change in DCIS detection was not statistically significant (from 1.3 to 3.2, p =0.250). For mammography alone, sensitivity was 24%; specificity, 89%; and PPV3, 25%. For the combination, sensitivity was 91% (p < 0.001); specificity, 83% (p < 0.001); and PPV3, 28% (p = 0.70). The recall rate increased from 11.0% with mammography alone to 17.6% (p < 0.001) for the combination; the biopsy rate increased from 1.3% for mammography alone to 4.2% (p < 0.001). CONCLUSION. When added to screening mammography, MBI performed using a radiopharmaceutical activity acceptable for screening (effective dose 2.4 mSv) yielded a supplemental cancer detection rate of 8.8 per 1000 women with mammographically dense breasts.

Quantitative imaging biomarkers: a review of statistical methods for computer algorithm comparisons.

Quantitative biomarkers from medical images are becoming important tools for clinical diagnosis, staging, monitoring, treatment planning, and development of new therapies. While there is a rich history of the development of quantitative imaging biomarker (QIB) techniques, little attention has been paid to the validation and comparison of the computer algorithms that implement the QIB measurements. In this paper we provide a framework for QIB algorithm comparisons. We first review and compare various study designs, including designs with the true value (e.g. phantoms, digital reference images, and zero-change studies), designs with a reference standard (e.g. studies testing equivalence with a reference standard), and designs without a reference standard (e.g. agreement studies and studies of algorithm precision). The statistical methods for comparing QIB algorithms are then presented for various study types using both aggregate and disaggregate approaches. We propose a series of steps for establishing the performance of a QIB algorithm, identify limitations in the current statistical literature, and suggest future directions for research.

Correlation between model observer and human observer performance in CT imaging when lesion location is uncertain.

PURPOSE: The purpose of this study was to investigate the correlation between model observer and human observer performance in CT imaging for the task of lesion detection and localization when the lesion location is uncertain. METHODS: Two cylindrical rods (3-mm and 5-mm diameters) were placed in a 35×26 cm torso-shaped water phantom to simulate lesions with -15 HU contrast at 120 kV. The phantom was scanned 100 times on a 128-slice CT scanner at each of four dose levels (CTDIvol=5.7, 11.4, 17.1, and 22.8 mGy). Regions of interest (ROIs) around each lesion were extracted to generate images with signal-present, with each ROI containing 128×128 pixels. Corresponding ROIs of signal-absent images were generated from images without lesion mimicking rods. The location of the lesion (rod) in each ROI was randomly distributed by moving the ROIs around each lesion. Human observer studies were performed by having three trained observers identify the presence or absence of lesions, indicating the lesion location in each image and scoring confidence for the detection task on a 6-point scale. The same image data were analyzed using a channelized Hotelling model observer (CHO) with Gabor channels. Internal noise was added to the decision variables for the model observer study. Area under the curve (AUC) of ROC and localization ROC (LROC) curves were calculated using a nonparametric approach. The Spearman's rank order correlation between the average performance of the human observers and the model observer performance was calculated for the AUC of both ROC and LROC curves for both the 3- and 5-mm diameter lesions. RESULTS: In both ROC and LROC analyses, AUC values for the model observer agreed well with the average values across the three human observers. The Spearman's rank order correlation values for both ROC and LROC analyses for both the 3- and 5-mm diameter lesions were all 1.0, indicating perfect rank ordering agreement of the figures of merit (AUC) between the average performance of the human observers and the model observer performance. CONCLUSIONS: In CT imaging of different sizes of low-contrast lesions (-15 HU), the performance of CHO with Gabor channels was highly correlated with human observer performance for the detection and localization tasks with uncertain lesion location in CT imaging at four clinically relevant dose levels. This suggests the ability of Gabor CHO model observers to meaningfully assess CT image quality for the purpose of optimizing scan protocols and radiation dose levels in detection and localization tasks for low-contrast lesions.

Comparison of radiologist performance with photon-counting full-field digital mammography to conventional full-field digital mammography.

RATIONALE AND OBJECTIVES: The purpose of this study was to assess the performance of a MicroDose photon-counting full-field digital mammography (PCM) system in comparison to full-field digital mammography (FFDM) for area under the receiver-operating characteristic (ROC) curve (AUC), sensitivity, specificity, and feature analysis of standard-view mammography for women presenting for screening mammography, diagnostic mammography, or breast biopsy. MATERIALS AND METHODS: A total of 133 women were enrolled in this study at two European medical centers, with 67 women who had a pre-existing 10-36 months FFDM enrolled prospectively into the study and 66 women who underwent breast biopsy and had screening PCM and diagnostic FFDM, including standard craniocaudal and mediolateral oblique views of the breast with the lesion, enrolled retrospectively. The case mix consisted of 49 cancers, 17 biopsy-benign cases, and 67 normal cases. Sixteen radiologists participated in the reader study and interpreted all 133 cases in both conditions, separated by washout period of ≥4 weeks. ROC curve and free-response ROC curve analyses were performed for noninferiority of PCM compared to FFDM using a noninferiority margin Δ value of 0.10. Feature analysis of the 66 cases with lesions was conducted with all 16 readers at the conclusion of the blinded reads. Mean glandular dose was recorded for all cases. RESULTS: The AUC for PCM was 0.947 (95% confidence interval [CI], 0.920-0.974) and for FFDM was 0.931 (95% CI, 0.898-0.964). Sensitivity per case for PCM was 0.936 (95% CI, 0.897-0.976) and for FFDM was 0.908 (95% CI, 0.856-0.960). Specificity per case for PCM was 0.764 (95% CI, 0.688-0.841) and for FFDM was 0.749 (95% CI, 0.668-0.830). Free-response ROC curve figures of merit were 0.920 (95% CI, 0.881-0.959) and 0.903 (95% CI, 0.858-0.948) for PCM and FFDM, respectively. Sensitivity per lesion was 0.903 (95% CI, 0.846-0.960) and 0.883 (95% CI, 0.823-0.944) for PCM and FFDM, respectively. The average false-positive marks per image of noncancer cases were 0.265 (95% CI, 0.171-0.359) and 0.281 (95% CI, 0.188-0.374) for PCM and FFDM, respectively. Noninferiority P values for AUC, sensitivity (per case and per lesion), specificity, and average false-positive marks per image were all statistically significant (P < .001). The noninferiority P value for free-response ROC was <.025, from the 95% CI for the difference. Feature analysis resulted in PCM being preferred to FFDM by the readers for ≥70% of the cases. The average mean glandular dose for PCM was 0.74 mGy (95% CI, 0.722-0.759 mGy) and for FFDM was 1.23 mGy (95% CI, 1.199-1.262 mGy). CONCLUSIONS: In this study, radiologist performance with PCM was not inferior to that with conventional FFDM at an average 40% lower mean glandular dose.

The National CT Colonography Trial: assessment of accuracy in participants 65 years of age and older.

PURPOSE: To conduct post-hoc analysis of National CT Colonography Trial data and compare the sensitivity and specificity of computed tomographic (CT) colonography in participants younger than 65 years with those in participants aged 65 years and older. MATERIALS AND METHODS: Of 2600 asymptomatic participants recruited at 15 centers for the trial, 497 were 65 years of age or older. Approval of this HIPAA-compliant study was obtained from the institutional review board of each site, and informed consent was obtained from each subject. Radiologists certified in CT colonography reported lesions 5 mm in diameter or larger. Screening detection of large (≥10-mm) histologically confirmed colorectal neoplasia was the primary end point; screening detection of smaller (6-9-mm) colorectal neoplasia was a secondary end point. The differences in sensitivity and specificity of CT colonography in the two age cohorts (age < 65 years and age ≥ 65 years) were estimated with bootstrap confidence intervals (CIs). RESULTS: Complete data were available for 477 participants 65 years of age or older (among 2531 evaluable participants). Prevalence of adenomas 1 cm or larger for the older participants versus the younger participants was 6.9% (33 of 477) versus 3.7% (76 of 2054) (P < .004). For large neoplasms, mean estimates for CT colonography sensitivity and specificity among the older cohort were 0.82 (95% CI: 0.644, 0.944) and 0.83 (95% CI: 0.779, 0.883), respectively. For large neoplasms in the younger group, CT colonography sensitivity and specificity were 0.92 (95% CI: 0.837, 0.967) and 0.86 (95% CI: 0.816, 0.899), respectively. Per-polyp sensitivity for large neoplasms for the older and younger populations was 0.75 (95% CI: 0.578, 0.869) and 0.84 (95% CI: 0.717, 0.924), respectively. For the older and younger groups, per-participant sensitivity was 0.72 (95% CI: 0.565, 0.854) and 0.81 (95% CI: 0.745, 0.882) for detecting adenomas 6 mm in diameter or larger. CONCLUSION: For most measures of diagnostic performance and in most subsets, the difference between senior-aged participants and those younger than 65 years was not statistically significant.

Evaluating imaging and computer-aided detection and diagnosis devices at the FDA.

This report summarizes the Joint FDA-MIPS Workshop on Methods for the Evaluation of Imaging and Computer-Assist Devices. The purpose of the workshop was to gather information on the current state of the science and facilitate consensus development on statistical methods and study designs for the evaluation of imaging devices to support US Food and Drug Administration submissions. Additionally, participants expected to identify gaps in knowledge and unmet needs that should be addressed in future research. This summary is intended to document the topics that were discussed at the meeting and disseminate the lessons that have been learned through past studies of imaging and computer-aided detection and diagnosis device performance.

Accuracy of CT colonography for detection of large adenomas and cancers.

BACKGROUND: Computed tomographic (CT) colonography is a noninvasive option in screening for colorectal cancer. However, its accuracy as a screening tool in asymptomatic adults has not been well defined. METHODS: We recruited 2600 asymptomatic study participants, 50 years of age or older, at 15 study centers. CT colonographic images were acquired with the use of standard bowel preparation, stool and fluid tagging, mechanical insufflation, and multidetector-row CT scanners (with 16 or more rows). Radiologists trained in CT colonography reported all lesions measuring 5 mm or more in diameter. Optical colonoscopy and histologic review were performed according to established clinical protocols at each center and served as the reference standard. The primary end point was detection by CT colonography of histologically confirmed large adenomas and adenocarcinomas (10 mm in diameter or larger) that had been detected by colonoscopy; detection of smaller colorectal lesions (6 to 9 mm in diameter) was also evaluated. RESULTS: Complete data were available for 2531 participants (97%). For large adenomas and cancers, the mean (+/-SE) per-patient estimates of the sensitivity, specificity, positive and negative predictive values, and area under the receiver-operating-characteristic curve for CT colonography were 0.90+/-0.03, 0.86+/-0.02, 0.23+/-0.02, 0.99+/-<0.01, and 0.89+/-0.02, respectively. The sensitivity of 0.90 (i.e., 90%) indicates that CT colonography failed to detect a lesion measuring 10 mm or more in diameter in 10% of patients. The per-polyp sensitivity for large adenomas or cancers was 0.84+/-0.04. The per-patient sensitivity for detecting adenomas that were 6 mm or more in diameter was 0.78. CONCLUSIONS: In this study of asymptomatic adults, CT colonographic screening identified 90% of subjects with adenomas or cancers measuring 10 mm or more in diameter. These findings augment published data on the role of CT colonography in screening patients with an average risk of colorectal cancer. (ClinicalTrials.gov number, NCT00084929; American College of Radiology Imaging Network [ACRIN] number, 6664.)

An evaluation of the variability of tumor-shape definition derived by experienced observers from CT images of supraglottic carcinomas (ACRIN protocol 6658).

PURPOSE: Accurate target definition is considered essential for sophisticated, image-guided radiation therapy; however, relatively little information has been reported that measures our ability to identify the precise shape of targets accurately. We decided to assess the manner in which eight "experts" interpreted the size and shape of tumors based on "real-life" contrast-enhanced computed tomographic (CT) scans. METHODS AND MATERIALS: Four neuroradiologists and four radiation oncologists (the authors) with considerable experience and presumed expertise in treating head-and-neck tumors independently contoured, slice-by-slice, his/her interpretation of the precise gross tumor volume (GTV) on each of 20 sets of CT scans taken from 20 patients who previously were enrolled in Radiation Therapy Oncology Group protocol 91-11. RESULTS: The average proportion of overlap (i.e., the degree of agreement) was 0.532 (95% confidence interval 0.457 to 0.606). There was a slight tendency for the proportion of overlap to increase with increasing average GTV. CONCLUSIONS: Our work suggests that estimation of tumor shape currently is imprecise, even for experienced physicians. In consequence, there appears to be a practical limit to the current trend of smaller fields and tighter margins.

Delayed calf muscle phosphocreatine recovery after exercise identifies peripheral arterial disease.

OBJECTIVES: In this study we intend to characterize phosphocreatine (PCr) recovery kinetics with phosphorus-31 ((31)P) magnetic resonance spectroscopy in symptomatic peripheral arterial disease (PAD) patients compared with control subjects and determine the diagnostic value and reproducibility of this parameter. BACKGROUND: Due to the inconsistent relationship between flow and function in PAD, novel techniques focused on the end-organ are needed to assess disease severity and measure therapeutic response. METHODS: Fourteen normal subjects (5 men, age 45 +/- 14 years) and 20 patients with mild-to-moderate symptomatic PAD (12 men, age 67 +/- 10 years, mean ankle brachial index 0.62 +/- 0.13) were studied. Subjects exercised one leg to exhaustion while supine in a 1.5-T magnetic resonance scanner using a custom-built plantar flexion device. Surface coil-localized, free induction decay acquisition localized to the mid-calf was used. Each 31P spectrum consisted of 25 signal averages at a repetition time of 550 ms. The PCr recovery time constant was calculated by monoexponential fit of PCr versus time, beginning at exercise completion. RESULTS: Median exercise time was 195.0 s in normal subjects and 162.5 s in PAD patients (p = 0.06). Despite shorter exercise times in patients, the median recovery time constant of PCr was 34.7 s in normal subjects and 91.0 s in PAD patients. Area under the receiver-operating characteristic curve was 0.925 +/- 0.045. Test-retest reliability was excellent. CONCLUSIONS: The PCr recovery time constant is prolonged in patients with symptomatic PAD compared with normal subjects. The method is reproducible and may be useful in the identification of disease. Further study of this parameter's ability to track response to therapy as well as its prognostic capability is warranted.

Interobserver reliability of computed tomography-derived primary tumor volume measurement in patients with supraglottic carcinoma.

BACKGROUND: Prior studies have determined that macroscopic ("gross") tumor volume (GTV), as calculated from pretreatment computer tomography (CT), was capable of predicting local control in squamous cell carcinoma arising in different subsites in the head and neck in patients who were treated with nonsurgical organ-preservation therapy. The majority of these studies were single-institution, retrospective investigations. Consequently, there has been concern that GTV measurements may not be reproducible by different readers at different institutions. The objective of the current study was to measure the interobserver reliability for GTV measurements for squamous cell carcinoma of the supraglottic larynx (SGSCCA) performed by different readers at different institutions. METHODS: Eight experienced readers (4 neuroradiologists and 4 radiation oncologists) from different institutions independently measured the pretreatment GTV of 20 patients with SGSCCA. The CT scans were obtained from patients entered into the definitive radiation therapy arm of Radiation Therapy Oncology Group protocol 91-11, who had supraglottic carcinoma and underwent pretreatment CT scans of the neck. Statistical analysis focused on interobserver reliability as measured by the intraclass correlation coefficient. RESULTS: The intraclass correlation coefficient was 0.81 (95% lower confidence bound, 0.71). This value was interpreted as "excellent." CONCLUSIONS: GTV measurements were reliable and reproducible when performed by neuroradiologists and radiation oncologists who were experienced in the interpretation of CT scans of the extracranial head and neck in patients with SGSCCA. The result implied that the correlation between GTV and local control should be reproducible across institutions.

Weaknesses of goodness-of-fit tests for evaluating propensity score models: the case of the omitted confounder.

PURPOSE: Propensity scores are used in observational studies to adjust for confounding, although they do not provide control for confounders omitted from the propensity score model. We sought to determine if tests used to evaluate logistic model fit and discrimination would be helpful in detecting the omission of an important confounder in the propensity score. METHODS: Using simulated data, we estimated propensity scores under two scenarios: (1) including all confounders and (2) omitting the binary confounder. We compared the propensity score model fit and discrimination under each scenario, using the Hosmer-Lemeshow goodness-of-fit (GOF) test and the c-statistic. We measured residual confounding in treatment effect estimates adjusted by the propensity score omitting the confounder. RESULTS: The GOF statistic and discrimination of propensity score models were the same for models excluding an important predictor of treatment compared to the full propensity score model. The GOF test failed to detect poor model fit for the propensity score model omitting the confounder. C-statistics under both scenarios were similar. Residual confounding was observed from using the propensity score excluding the confounder (range: 1-30%). CONCLUSIONS: Omission of important confounders from the propensity score leads to residual confounding in estimates of treatment effect. However, tests of GOF and discrimination do not provide information to detect missing confounders in propensity score models. Our findings suggest that it may not be necessary to compute GOF statistics or model discrimination when developing propensity score models.

Principles for modeling propensity scores in medical research: a systematic literature review.

PURPOSE: To document which established criteria for logistic regression modeling researchers consider when using propensity scores in observational studies. METHODS: We performed a systematic review searching Medline and Science Citation to identify observational studies published in 2001 that addressed clinical questions using propensity score methods to adjust for treatment assignment. We abstracted aspects of propensity score model development (e.g. variable selection criteria, continuous variables included in correct functional form, interaction inclusion criteria), model discrimination and goodness of fit for 47 studies meeting inclusion criteria. RESULTS: We found few studies reporting on the propensity score model development or evaluation of model fit. CONCLUSIONS: Reporting of aspects related to propensity score model development is limited and raises questions about the value of these principles in developing propensity scores from which unbiased treatment effects are estimated.

Three methods for analysing correlated ROC curves: a comparison in real data sets from multi-reader, multi-case studies with a factorial design.

This paper compares three published methods for analysing multiple correlated ROC curves: a method using generalized estimating equations with marginal non-proportional ordinal regression models; a method using jackknifed pseudovalues of summary statistics; a method using a corrected F-test from analysis of variance of summary statistics. Use of these methods is illustrated through six real data examples from studies with the common factorial design, that is, multiple readers interpreting images obtained with each test modality on each study subject. The issue of the difference between typical summary statistics and summary statistics from typical ROC curves is explored. The examples also address similarities and differences among the analytical methods. In particular, while point estimates of differences between test modalities are similar, the standard errors of these differences do not agree for all three methods. A simulation study supports the standard errors provided by the generalized estimating equations with marginal non-proportional ordinal regression models.

Computerized tomographic colonography: performance evaluation in a retrospective multicenter setting.

BACKGROUND & AIMS: No multicenter study has been reported evaluating the performance and interobserver variability of computerized tomographic colonography. The aim of this study was to assess the accuracy of computerized tomographic colonography for detecting clinically important colorectal neoplasia (polyps >or=10 mm in diameter) in a multi-institutional study. METHODS: A retrospective study was developed from 341 patients who had computerized tomographic colonography and colonoscopy among 8 medical centers. Colonoscopy and pathology reports provided the standard. A random sample of 117 patients, stratified by criterion standard, was requested. Ninety-three patients were included (47% with polyps >or=10 mm; mean age, 62 years; 56% men; 84% white; 40% reported colorectal symptoms; 74% at increased risk for colorectal cancer). Eighteen radiologists blinded to the criterion standard interpreted computerized tomography colonography examinations, each using 2 of 3 different software display platforms. RESULTS: The average area under the receiver operating characteristic curve for identifying patients with at least 1 lesion >or=10 mm was 0.80 (95% lower confidence bound, 0.74). The average sensitivity and specificity were 75% (95% lower confidence bound, 68%) and 73% (95% lower confidence bound, 66%), respectively. Per-polyp sensitivity was 75%. A trend was observed for better performance with more observer experience. There was no difference in performance across software display platforms. CONCLUSIONS: Computerized tomographic colonography performance compared favorably with reported performance of fecal occult blood testing, flexible sigmoidoscopy, and barium enema. A prospective study evaluating the performance of computerized tomography colonography in a screening population is indicated.