Corticosteroids to prevent kidney scarring in children with a febrile urinary tract infection: a randomized trial.

BACKGROUND: To evaluate the efficacy of adjuvant systemic corticosteroids in reducing kidney scarring. A previous study suggested that use of adjuvant systemic corticosteroids reduces kidney scarring in children radiologically confirmed to have extensive pyelonephritis. Efficacy of corticosteroids for children with febrile urinary tract infection (UTI) has not been studied. METHODS: Children aged 2 months to 6 years with their first febrile UTI were randomized to corticosteroids or placebo for 3 days (both arms received antimicrobial therapy); kidney scarring was assessed using 99mTc-dimercaptosuccinic acid kidney scan 5-24 months after the initial UTI. RESULTS: We randomized 546 children of which 385 had a UTI and 254 had outcome kidney scans (instead of the 320 planned). Rates of kidney scarring were 9.8% (12/123) and 16.8% (22/131) in the corticosteroid and placebo groups, respectively (p = 0.16), corresponding to an absolute risk reduction of 5.9% (95% confidence interval: - 2.2, 14.1). CONCLUSION: While children randomized to adjuvant corticosteroids tended to develop fewer kidney scars than children who were randomized to receive placebo, a statistically significant difference was not achieved. However, the study was limited by not reaching its intended sample size. CLINICAL TRIAL REGISTRATION: Clinicaltrials.gov , NCT01391793, Registered 7/12/2011 Graphical abstract.

Shortened Antimicrobial Treatment for Acute Otitis Media in Young Children.

BACKGROUND: Limiting the duration of antimicrobial treatment constitutes a potential strategy to reduce the risk of antimicrobial resistance among children with acute otitis media. METHODS: We assigned 520 children, 6 to 23 months of age, with acute otitis media to receive amoxicillin-clavulanate either for a standard duration of 10 days or for a reduced duration of 5 days followed by placebo for 5 days. We measured rates of clinical response (in a systematic fashion, on the basis of signs and symptomatic response), recurrence, and nasopharyngeal colonization, and we analyzed episode outcomes using a noninferiority approach. Symptom scores ranged from 0 to 14, with higher numbers indicating more severe symptoms. RESULTS: Children who were treated with amoxicillin-clavulanate for 5 days were more likely than those who were treated for 10 days to have clinical failure (77 of 229 children [34%] vs. 39 of 238 [16%]; difference, 17 percentage points [based on unrounded data]; 95% confidence interval, 9 to 25). The mean symptom scores over the period from day 6 to day 14 were 1.61 in the 5-day group and 1.34 in the 10-day group (P=0.07); the mean scores at the day-12-to-14 assessment were 1.89 versus 1.20 (P=0.001). The percentage of children whose symptom scores decreased more than 50% (indicating less severe symptoms) from baseline to the end of treatment was lower in the 5-day group than in the 10-day group (181 of 227 children [80%] vs. 211 of 233 [91%], P=0.003). We found no significant between-group differences in rates of recurrence, adverse events, or nasopharyngeal colonization with penicillin-nonsusceptible pathogens. Clinical-failure rates were greater among children who had been exposed to three or more children for 10 or more hours per week than among those with less exposure (P=0.02) and were also greater among children with infection in both ears than among those with infection in one ear (P<0.001). CONCLUSIONS: Among children 6 to 23 months of age with acute otitis media, reduced-duration antimicrobial treatment resulted in less favorable outcomes than standard-duration treatment; in addition, neither the rate of adverse events nor the rate of emergence of antimicrobial resistance was lower with the shorter regimen. (Funded by the National Institute of Allergy and Infectious Diseases and the National Center for Research Resources; ClinicalTrials.gov number, NCT01511107 .).

Reduced-Concentration Clavulanate for Young Children with Acute Otitis Media.

Amoxicillin-clavulanate (A/C) is currently the most effective oral antimicrobial in treating children with acute otitis media (AOM), but the standard dosage of 90 mg amoxicillin/6.4 mg clavulanate/kg of body weight/day commonly causes diarrhea. We examined whether an A/C formulation containing lower concentrations of clavulanate would result in less diarrhea while maintaining plasma levels of amoxicillin and clavulanate adequate to eradicate middle-ear pathogens and to achieve clinical success. We conducted an open-label study in children with AOM who were 6 to 23 months of age. In phase 1, we treated 40 children with a reduced-clavulanate A/C formulation providing 90 mg amoxicillin/3.2 mg clavulanate/kg/day for 10 days. In phase 2, we treated 72 children with the same formulation at a dosage of 80 mg amoxicillin/2.85 mg clavulanate/kg/day for 10 days. We compared the rates of protocol-defined diarrhea (PDD), diaper dermatitis, and AOM clinical response in these children with rates we had reported in children who received the standard A/C regimen, and we obtained plasma levels of amoxicillin and clavulanate at various time points. Outcomes in phase 1 children and in children who had received the standard regimen did not differ significantly. Rates of PDD in children receiving phase 2 and standard regimens were 17% and 26%, respectively (P = 0.10). The corresponding rates of diaper dermatitis were 21% and 33% (P = 0.04) and of AOM treatment failure were 12% and 16% (P = 0.44). Symptomatic responses did not differ significantly between regimens; both gave clavulanate levels sufficient to inhibit ß-lactamase activity. In young children with AOM, clavulanate dosages lower than those currently used may be associated with fewer side effects without reducing clinical efficacy. (This study has been registered at ClinicalTrials.gov under registration no. NCT02630992.).

Treatment of acute otitis media in children under 2 years of age.

BACKGROUND: Recommendations vary regarding immediate antimicrobial treatment versus watchful waiting for children younger than 2 years of age with acute otitis media. METHODS: We randomly assigned 291 children 6 to 23 months of age, with acute otitis media diagnosed with the use of stringent criteria, to receive amoxicillin-clavulanate or placebo for 10 days. We measured symptomatic response and rates of clinical failure. RESULTS: Among the children who received amoxicillin-clavulanate, 35% had initial resolution of symptoms by day 2, 61% by day 4, and 80% by day 7; among children who received placebo, 28% had initial resolution of symptoms by day 2, 54% by day 4, and 74% by day 7 (P=0.14 for the overall comparison). For sustained resolution of symptoms, the corresponding values were 20%, 41%, and 67% with amoxicillin-clavulanate, as compared with 14%, 36%, and 53% with placebo (P=0.04 for the overall comparison). Mean symptom scores over the first 7 days were lower for the children treated with amoxicillin-clavulanate than for those who received placebo (P=0.02). The rate of clinical failure--defined as the persistence of signs of acute infection on otoscopic examination--was also lower among the children treated with amoxicillin-clavulanate than among those who received placebo: 4% versus 23% at or before the visit on day 4 or 5 (P<0.001) and 16% versus 51% at or before the visit on day 10 to 12 (P<0.001). Mastoiditis developed in one child who received placebo. Diarrhea and diaper-area dermatitis were more common among children who received amoxicillin-clavulanate. There were no significant changes in either group in the rates of nasopharyngeal colonization with nonsusceptible Streptococcus pneumoniae. CONCLUSIONS: Among children 6 to 23 months of age with acute otitis media, treatment with amoxicillin-clavulanate for 10 days tended to reduce the time to resolution of symptoms and reduced the overall symptom burden and the rate of persistent signs of acute infection on otoscopic examination. (Funded by the National Institute of Allergy and Infectious Diseases; ClinicalTrials.gov number, NCT00377260.).

Rituximab in the treatment of refractory adult and juvenile dermatomyositis and adult polymyositis: a randomized, placebo-phase trial.

OBJECTIVE: To assess the safety and efficacy of rituximab in a randomized, double-blind, placebo-phase trial in adult and pediatric myositis patients. METHODS: Adults with refractory polymyositis (PM) and adults and children with refractory dermatomyositis (DM) were enrolled. Entry criteria included muscle weakness and ≥2 additional abnormal values on core set measures (CSMs) for adults. Juvenile DM patients required ≥3 abnormal CSMs, with or without muscle weakness. Patients were randomized to receive either rituximab early or rituximab late, and glucocorticoid or immunosuppressive therapy was allowed at study entry. The primary end point compared the time to achieve the International Myositis Assessment and Clinical Studies Group preliminary definition of improvement (DOI) between the 2 groups. The secondary end points were the time to achieve ≥20% improvement in muscle strength and the proportions of patients in the early and late rituximab groups achieving the DOI at week 8. RESULTS: Among 200 randomized patients (76 with PM, 76 with DM, and 48 with juvenile DM), 195 showed no difference in the time to achieving the DOI between the rituximab late (n = 102) and rituximab early (n = 93) groups (P = 0.74 by log rank test), with a median time to achieving a DOI of 20.2 weeks and 20.0 weeks, respectively. The secondary end points also did not significantly differ between the 2 treatment groups. However, 161 (83%) of the randomized patients met the DOI, and individual CSMs improved in both groups throughout the 44-week trial. CONCLUSION: Although there were no significant differences in the 2 treatment arms for the primary and secondary end points, 83% of adult and juvenile myositis patients with refractory disease met the DOI. The role of B cell-depleting therapies in myositis warrants further study, with consideration for a different trial design.

On use of partial area under the ROC curve for evaluation of diagnostic performance.

Evaluation of diagnostic performance is a necessary component of new developments in many fields including medical diagnostics and decision making. The methodology for statistical analysis of diagnostic performance continues to develop, offering new analytical tools for conventional inferences and solutions for novel and increasingly more practically relevant questions. In this paper, we focus on the partial area under the Receiver Operating Characteristic (ROC) curve or pAUC. This summary index is considered to be more practically relevant than the area under the entire ROC curve (AUC), but because of several perceived limitations, it is not used as often. To improve interpretation, results for pAUC analysis are frequently reported using a rescaled index such as the standardized partial AUC proposed by McClish (1989). We derive two important properties of the relationship between the 'standardized' pAUC and the defined range of interest, which could facilitate a wider and more appropriate use of this important summary index. First, we mathematically prove that the 'standardized' pAUC increases with increasing range of interest for practically common ROC curves. Second, using comprehensive numerical investigations, we demonstrate that, contrary to common belief, the uncertainty about the estimated standardized pAUC can either decrease or increase with an increasing range of interest. Our results indicate that the partial AUC could frequently offer advantages in terms of statistical uncertainty of the estimation. In addition, selection of a wider range of interest will likely lead to an increased estimate even for standardized pAUC.

Association Between Nasopharyngeal Colonization and Clinical Outcome in Children With Acute Otitis Media.

BACKGROUND: Young children with acute otitis media (AOM) frequently exhibit nasopharyngeal colonization with either Streptococcus pneumoniae, Haemophilus influenzae or both pathogens. We aimed to determine if antibiotics could be spared or shortened in those without nasopharyngeal colonization with either pathogen. METHODS: In 2 separate randomized clinical trials in children aged 6-23 months with stringently-diagnosed AOM, we performed bacterial cultures on nasopharyngeal specimens collected at the time of diagnosis. In the first trial, we compared the efficacy of amoxicillin/clavulanate (amox/clav) administered for 10 days vs. that of placebo, and in the second trial, we compared the efficacy of amox/clav administered for 10 days vs. 5 days. In each trial, we classified children as being colonized with both S. pneumoniae and H. influenzae, S. pneumoniae alone, H. influenzae alone, or neither pathogen, and as experiencing either clinical success or clinical failure at the end-of-therapy visit, based on previously reported a priori criteria. RESULTS: We evaluated 796 children. Among children randomized to amox/clav, those colonized with either S. pneumoniae or H. influenzae or both were approximately twice as likely to experience clinical failure as children not colonized with either pathogen (odds ratio: 1.8; confidence intervals: 1.2-2.9). In contrast, among children randomized to placebo, clinical failure at the end-of-therapy visit was not associated with nasopharyngeal culture results at the time of diagnosis. CONCLUSIONS: Children colonized with either S. pneumoniae or H. influenzae or both have a greater chance of treatment failure than children colonized with neither pathogen.

Identifying children with vesicoureteral reflux: a comparison of 2 approaches.

PURPOSE: Various screening approaches have been proposed to identify the subgroup of children with urinary tract infection who have vesicoureteral reflux. However, few studies have compared the sensitivity of screening approaches in a representative population of young children. We compared the sensitivities of the top-down ((99m)technetium dimercaptosuccinic acid renal scan to screen) and biomarker based (C-reactive protein level at presentation) approaches in identifying children with vesicoureteral reflux. MATERIALS AND METHODS: We calculated the sensitivity of the 2 screening approaches in detecting vesicoureteral reflux and subsequently high grade (III or greater) vesicoureteral reflux in children. RESULTS: The top-down and C-reactive protein based approaches missed 33% and 29% of cases of high grade vesicoureteral reflux, respectively. CONCLUSIONS: The sensitivity of the top-down approach for detecting high grade vesicoureteral reflux was lower than previously reported. Further study of novel methods to identify children at risk for renal scarring is warranted.

Randomized Trial of Tocilizumab in the Treatment of Refractory Adult Polymyositis and Dermatomyositis.

OBJECTIVE: To assess the efficacy and tolerability of tocilizumab in a multicenter, randomized, double-blind, placebo-controlled trial in refractory adult patients with dermatomyositis (DM) and polymyositis (PM). METHODS: Thirty-six subjects with probable or definite DM/PM were enrolled in a 6-month phase 2B clinical trial and randomized 1:1 to receive tocilizumab (8 mg/kg intravenously) or placebo every 4 weeks for 24 weeks. Eligible subjects had either a DM rash, a myositis-associated autoantibody or an adjudicated PM diagnosis. Active disease was defined by at least three of six abnormal core set measures (CSMs), including a manual muscle testing (MMT)-8 score of less than 136/150. If the MMT-8 score was greater than 136, then a cutaneous score of 3 or more (10 cm visual analogue scale) was required along with three additional abnormal CSMs indicating disease activity. The primary endpoint compared the Total Improvement Score (TIS) between both arms from week 4 to 24. Secondary outcomes included time to meeting minimal TIS improvement, changes in CSMs, time to worsening, steroid-sparing effect, proportion of subjects meeting more stringent improvement criteria, and safety outcomes. RESULTS: There was no significant difference (P = 0.86) in the TIS over 24 weeks between tocilizumab and placebo arms. The secondary endpoints of time to improvement (minimal, moderate, or major), time to worsening, CSM changes, safety outcomes, and steroid-sparing effect were also not significantly different between arms. CONCLUSION: Tocilizumab was safe and well tolerated but did not meet the primary or secondary efficacy outcomes in refractory DM and PM in this 24-week phase 2B study.

Localized detection and classification of abnormalities on FFDM and tomosynthesis examinations rated under an FROC paradigm.

OBJECTIVE: The purpose of our study was to assess diagnostic performance when retrospectively interpreting full-field digital mammography (FFDM) and breast tomosynthesis examinations under a free-response receiver operating characteristic (FROC) paradigm. MATERIALS AND METHODS: We performed FROC analysis of a previously reported study in which eight experienced radiologists interpreted 125 examinations, including 35 with verified cancers. The FROC paradigm involves detecting, locating, and rating each suspected abnormality. Radiologists reviewed and rated both FFDM alone and a combined display mode of FFDM and digital breast tomosynthesis (DBT) (combined). Observer performance levels were assessed and compared with respect to the fraction of correctly identified abnormalities, the number of reported location-specific findings (both true and false), and their associated ratings. The analysis accounts for the number and locations of findings and the location-based ratings using a summary performance index (Λ), which is the FROC analog of the area between the receiver operating characteristic curve and the diagonal (chance) line. RESULTS: Under the FROC paradigm, each reader detected more true abnormalities associated with cancer, or a higher true-positive fraction, under the combined mode. In an analysis focused on both the number of findings and associated location-based ratings, each of the radiologists performed better under the combined mode compared with FFDM alone, with increases in Λ ranging from 5% to 34%. On average, under the combined mode radiologists achieved a 16% improvement in Λ compared with the FFDM alone mode (95% CI, 7-26%; p < 0.01). CONCLUSION: We showed that DBT-based breast imaging in combination with FFDM could result in better performance under the FROC paradigm.

Tympanostomy tubes and developmental outcomes at 9 to 11 years of age.

BACKGROUND: Developmental impairments in children have been attributed to persistent middle-ear effusion in their early years of life. Previously, we reported that among children younger than 3 years of age with persistent middle-ear effusion, prompt as compared with delayed insertion of tympanostomy tubes did not result in improved cognitive, language, speech, or psychosocial development at 3, 4, or 6 years of age. However, other important components of development could not be assessed until the children were older. METHODS: We enrolled 6350 infants soon after birth and evaluated them regularly for middle-ear effusion. Before 3 years of age, 429 children with persistent effusion were randomly assigned to undergo the insertion of tympanostomy tubes either promptly or up to 9 months later if effusion persisted. We assessed literacy, attention, social skills, and academic achievement in 391 of these children at 9 to 11 years of age. RESULTS: Mean (+/-SD) scores on 48 developmental measures in the group of children who were assigned to undergo early insertion of tympanostomy tubes did not differ significantly from the scores in the group that was assigned to undergo delayed insertion. These measures included the Passage Comprehension subtest of the Woodcock Reading Mastery Tests (mean score, 98+/-12 in the early-treatment group and 99+/-12 in the delayed-treatment group); the Spelling, Writing Samples, and Calculation subtests of the Woodcock-Johnson III Tests of Achievement (96+/-13 and 97+/-16; 104+/-14 and 105+/-15; and 99+/-13 and 99+/-13, respectively); and inattention ratings on visual and auditory continuous performance tests. CONCLUSIONS: In otherwise healthy young children who have persistent middle-ear effusion, as defined in our study, prompt insertion of tympanostomy tubes does not improve developmental outcomes up to 9 to 11 years of age. (ClinicalTrials.gov number, NCT00365092 [ClinicalTrials.gov].).

Use of likelihood ratios for comparisons of binary diagnostic tests: underlying ROC curves.

PURPOSE: When comparing binary test results from two diagnostic systems, superiority in both "sensitivity" and "specificity" also implies differences in all conventional summary indices and locally in the underlying receiver operating characteristics (ROC) curves. However, when one of the two binary tests has higher sensitivity and lower specificity (or vice versa), comparisons of their performance levels are nontrivial and the use of different summary indices may lead to contradictory conclusions. A frequently used approach that is free of subjectivity associated with summary indices is based on the comparison of the underlying ROC curves that requires the collection of rating data using multicategory scales, whether natural or experimentally imposed. However, data for reliable estimation of ROC curves are frequently unavailable. The purpose of this article is to develop an approach of using "diagnostic likelihood ratios", namely, likelihood ratios of "positive" or "negative" responses, to make simple inferences regarding the underlying ROC curves and associated areas in the absence of reliable rating data or regarding the relative binary characteristics, when these are of primary interest. METHODS: For inferences related to underlying curves, the authors exploit the assumption of concavity of the true underlying ROC curve to describe conditions under which these curves have to be different and under which the curves have different areas. For scenarios when the binary characteristics are of primary interest, the authors use characteristics of "chance performance" to demonstrate that the derived conditions provide strong evidence of superiority of one binary test as compared to another. By relating these derived conditions to hypotheses about the true likelihood ratios of two binary diagnostic tests being compared, the authors enable a straightforward statistical procedure for the corresponding inferences. RESULTS: The authors derived simple algebraic and graphical methods for describing the conditions for superiority of one of two diagnostic tests with respect to their binary characteristics, the underlying ROC curves, or the areas under the curves. The graphical regions are useful for identifying potential differences between two systems, which then have to be tested statistically. The simple statistical tests can be performed with well known methods for comparison of diagnostic likelihood ratios. The developed approach offers a solution for some of the more difficult to analyze scenarios, where diagnostic tests do not demonstrate concordant differences in terms of both sensitivity and specificity. In addition, the resulting inferences do not contradict the conclusions that can be obtained using conventional and reasonably defined summary indices. CONCLUSIONS: When binary diagnostic tests are of primary interest, the proposed approach offers an objective and powerful method for comparing two binary diagnostic tests. The significant advantage of this method is that it enables objective analyses when one test has higher sensitivity but lower specificity, while ensuring agreement with study conclusions based on other reasonable and widely acceptable summary indices. For truly multicategory diagnostic tests, the proposed method can help in concluding inferiority of one of the diagnostic tests based on binary data, thereby potentially saving the need for conducting a more expensive multicategory ROC study.

Responsiveness and construct validity of a symptom scale for acute otitis media.

BACKGROUND: Because resolution of symptoms is a primary goal of antimicrobial therapy in children with acute otitis media (AOM), measurement of symptoms in studies of antimicrobial effectiveness in such children is important. We have developed a scale for measuring symptoms of AOM in young children (AOM-SOS), and we present data on its construct validity and responsiveness. METHODS: We followed children 3 months to 3 years of age with AOM, who were receiving antimicrobial treatment, using the AOM-SOS scale. The scale was administered at the enrollment visit, as a twice-a-day diary measure, and at the follow-up visit (days 5-7). To evaluate construct validity, we examined the correlation, at entry, between AOM-SOS scores and scores on other measures of pain and functional status. To evaluate the scale's responsiveness, we examined the change in scale scores from entry to follow-up. We also examined the levels of agreement between the scale scores and overall assessments of the children by parents. RESULTS: We enrolled 70 children (mean age 12.5 months) of whom 57 returned for follow-up. The magnitude of the correlations between the AOM-SOS scale scores and other measures of pain and functional status ranged from 0.56 to 0.84. The responsiveness of the AOM-SOS, as measured by the standardized response mean was 1.20. CONCLUSIONS: These data support the validity and responsiveness of the AOM-SOS; the scale seems to measure effectively both pain and overall functional status in young children with AOM. Changes in score over the first few days of illness were substantial and generally matched the assessments both of parents and of clinicians. The AOM-SOS promises to be useful as an outcome measure in clinical studies of AOM.

Area under the free-response ROC curve (FROC) and a related summary index.

Free-response assessment of diagnostic systems continues to gain acceptance in areas related to the detection, localization, and classification of one or more "abnormalities" within a subject. A free-response receiver operating characteristic (FROC) curve is a tool for characterizing the performance of a free-response system at all decision thresholds simultaneously. Although the importance of a single index summarizing the entire curve over all decision thresholds is well recognized in ROC analysis (e.g., area under the ROC curve), currently there is no widely accepted summary of a system being evaluated under the FROC paradigm. In this article, we propose a new index of the free-response performance at all decision thresholds simultaneously, and develop a nonparametric method for its analysis. Algebraically, the proposed summary index is the area under the empirical FROC curve penalized for the number of erroneous marks, rewarded for the fraction of detected abnormalities, and adjusted for the effect of the target size (or "acceptance radius"). Geometrically, the proposed index can be interpreted as a measure of average performance superiority over an artificial "guessing" free-response process and it represents an analogy to the area between the ROC curve and the "guessing" or diagonal line. We derive the ideal bootstrap estimator of the variance, which can be used for a resampling-free construction of asymptotic bootstrap confidence intervals and for sample size estimation using standard expressions. The proposed procedure is free from any parametric assumptions and does not require an assumption of independence of observations within a subject. We provide an example with a dataset sampled from a diagnostic imaging study and conduct simulations that demonstrate the appropriateness of the developed procedure for the considered sample sizes and ranges of parameters.

Developmental outcomes after early or delayed insertion of tympanostomy tubes.

BACKGROUND: To prevent later developmental impairments, myringotomy with the insertion of tympanostomy tubes has often been undertaken in young children who have persistent otitis media with effusion. We previously reported that prompt as compared with delayed insertion of tympanostomy tubes in children with persistent effusion who were younger than three years of age did not result in improved developmental outcomes at three or four years of age. However, the effect on the outcomes of school-age children is unknown. METHODS: We enrolled 6350 healthy infants younger than 62 days of age and evaluated them regularly for middle-ear effusion. Before three years of age, 429 children with persistent middle-ear effusion were randomly assigned to have tympanostomy tubes inserted either promptly or up to nine months later if effusion persisted. We assessed developmental outcomes in 395 of these children at six years of age. RESULTS: At six years of age, 85 percent of children in the early-treatment group and 41 percent in the delayed-treatment group had received tympanostomy tubes. There were no significant differences in mean (+/-SD) scores favoring early versus delayed treatment on any of 30 measures, including the Wechsler Full-Scale Intelligence Quotient (98+/-13 vs. 98+/-14); Number of Different Words test, a measure of word diversity (183+/-36 vs. 175+/-36); Percentage of Consonants Correct-Revised test, a measure of speech-sound production (96+/-2 vs. 96+/-3); the SCAN test, a measure of central auditory processing (95+/-15 vs. 96+/-14); and several measures of behavior and emotion. CONCLUSIONS: In otherwise healthy children younger than three years of age who have persistent middle-ear effusion within the duration of effusion that we studied, prompt insertion of tympanostomy tubes does not improve developmental outcomes at six years of age.

Comparing areas under receiver operating characteristic curves: potential impact of the "Last" experimentally measured operating point.

A specific issue related to the selection of the analytic tool used when comparing the estimated performance of systems within the receiver operating characteristic (ROC) paradigm is reviewed. This issue is related to the possible effect of the last experimentally ascertained ROC data point in terms of highest true-positive and false-positive fractions. An example of a case is presented where the selection of a specific analysis approach could affect the study conclusion from being not statistically significant for parametric analysis and significant for nonparametric analysis. This is followed by recommendations that should help prevent misinterpretation of results.

The "laboratory" effect: comparing radiologists' performance and variability during prospective clinical and laboratory mammography interpretations.

PURPOSE: To compare radiologists' performance during interpretation of screening mammograms in the clinic with their performance when reading the same mammograms in a retrospective laboratory study. MATERIALS AND METHODS: This study was conducted under an institutional review board-approved, HIPAA-compliant protocol; the need for informed consent was waived. Nine experienced radiologists rated an enriched set of mammograms that they had personally read in the clinic (the "reader-specific" set) mixed with an enriched "common" set of mammograms that none of the participants had previously read in the clinic by using a screening Breast Imaging Reporting and Data System (BI-RADS) rating scale. The original clinical recommendations to recall the women for a diagnostic work-up, for both reader-specific and common sets, were compared with their recommendations during the retrospective experiment. The results are presented in terms of reader-specific and group-averaged sensitivity and specificity levels and the dispersion (spread) of reader-specific performance estimates. RESULTS: On average, the radiologists' performance was significantly better in the clinic than in the laboratory (P = .035). Interreader dispersion of the computed performance levels was significantly lower during the clinical interpretations (P < .01). CONCLUSION: Retrospective laboratory experiments may not represent either expected performance levels or interreader variability during clinical interpretations of the same set of mammograms in the clinical environment well.

On comparing methods for discriminating between actually negative and actually positive subjects with FROC type data.

The task of searching and detecting multiple abnormalities depicted on an image, or a series of images, is a common problem in different areas such as military target detection or diagnostic medical imaging. A free response receiver operating characteristic (FROC) approach for assessing performance in many of these scenarios entails marking the locations of suspected abnormalities and indicating a level of suspicion at each of the marked locations. One of the important characteristics of a system being evaluated under the FROC paradigm is its performance in the conventional ROC domain, namely classifying a subject (or a unit of interest) as "negative" or "positive" in regard to the presence of the abnormality (or any of the abnormalities) of interest. With FROC data we can compare subjects by specifying a function of multiple scores within a subject. This approach allows formulating subject-based ROC type indices that can be estimated using existing ROC concepts. In this article we focus on indices that reflect the ability of the system to discriminate between actually negative and actually positive subjects. We consider a previously proposed index that is based on the comparison of the highest scores on subjects and two new indices that are based on potentially more stable comparison functions, namely comparison of average scores and stochastic dominance. Based on these indices we develop nonparametric procedures for comparing subject-based discriminative ability of diagnostic systems being evaluated under the FROC paradigm. We also investigate the properties of the statistical procedures in a simulation study.

Binary and multi-category ratings in a laboratory observer performance study: a comparison.

The authors investigated radiologists, performances during retrospective interpretation of screening mammograms when using a binary decision whether to recall a woman for additional procedures or not and compared it with their receiver operating characteristic (ROC) type performance curves using a semi-continuous rating scale. Under an Institutional Review Board approved protocol nine experienced radiologists independently rated an enriched set of 155 examinations that they had not personally read in the clinic, mixed with other enriched sets of examinations that they had individually read in the clinic, using both a screening BI-RADS rating scale (recall/not recall) and a semi-continuous ROC type rating scale (0 to 100). The vertical distance, namely the difference in sensitivity levels at the same specificity levels, between the empirical ROC curve and the binary operating point were computed for each reader. The vertical distance averaged over all readers was used to assess the proximity of the performance levels under the binary and ROC-type rating scale. There does not appear to be any systematic tendency of the readers towards a better performance when using either of the two rating approaches, namely four readers performed better using the semi-continuous rating scale, four readers performed better with the binary scale, and one reader had the point exactly on the empirical ROC curve. Only one of the nine readers had a binary "operating point" that was statistically distant from the same reader's empirical ROC curve. Reader-specific differences ranged from -0.046 to 0.128 with an average width of the corresponding 95% confidence intervals of 0.2 and p-values ranging for individual readers from 0.050 to 0.966. On average, radiologists performed similarly when using the two rating scales in that the average distance between the run in individual reader's binary operating point and their ROC curve was close to zero. The 95% confidence interval for the fixed-reader average (0.016) was (-0.0206, 0.0631) (two-sided p-value 0.35). In conclusion the authors found that in retrospective observer performance studies the use of a binary response or a semi-continuous rating scale led to consistent results in terms of performance as measured by sensitivity-specificity operating points.