Histopathological diagnosis of cutaneous melanocytic lesions: blinded and nonblinded second opinions offer similar improvement in diagnostic accuracy.

BACKGROUND: Previous studies of second opinions in the diagnosis of melanocytic skin lesions have examined blinded second opinions, which do not reflect usual clinical practice. The current study, conducted in the USA, investigated both blinded and nonblinded second opinions for their impact on diagnostic accuracy. METHODS: In total, 100 melanocytic skin biopsy cases, ranging from benign to invasive melanoma, were interpreted by 74 dermatopathologists. Subsequently, 151 dermatopathologists performed nonblinded second and third reviews. We compared the accuracy of single reviewers, second opinions obtained from independent, blinded reviewers and second opinions obtained from sequential, nonblinded reviewers. Accuracy was defined with respect to a consensus reference diagnosis. RESULTS: The mean case-level diagnostic accuracy of single reviewers was 65.3% (95% CI 63.4-67.2%). Second opinions arising from sequential, nonblinded reviewers significantly improved accuracy to 69.9% (95% CI 68.0-71.7%; P < 0.001). Similarly, second opinions arising from blinded reviewers improved upon the accuracy of single reviewers (69.2%; 95% CI 68.0-71.7%). Nonblinded reviewers were more likely than blinded reviewers to give diagnoses in the same diagnostic classes as the first diagnosis. Nonblinded reviewers tended to be more confident when they agreed with previous reviewers, even with inaccurate diagnoses. CONCLUSION: We found that both blinded and nonblinded second reviewers offered a similar modest improvement in diagnostic accuracy compared with single reviewers. Obtaining second opinions with knowledge of previous reviews tends to generate agreement among reviews, and may generate unwarranted confidence in an inaccurate diagnosis. Combining aspects of both blinded and nonblinded review in practice may leverage the advantages while mitigating the disadvantages of each approach. Specifically, a second pathologist could give an initial diagnosis blinded to the results of the first pathologist, with subsequent nonblinded discussion between the two pathologists if their diagnoses differ.

Combining biomarkers by maximizing the true positive rate for a fixed false positive rate.

Biomarkers abound in many areas of clinical research, and often investigators are interested in combining them for diagnosis, prognosis, or screening. In many applications, the true positive rate (TPR) for a biomarker combination at a prespecified, clinically acceptable false positive rate (FPR) is the most relevant measure of predictive capacity. We propose a distribution-free method for constructing biomarker combinations by maximizing the TPR while constraining the FPR. Theoretical results demonstrate desirable properties of biomarker combinations produced by the new method. In simulations, the biomarker combination provided by our method demonstrated improved operating characteristics in a variety of scenarios when compared with alternative methods for constructing biomarker combinations. Thus, use of our method could lead to the development of better biomarker combinations, increasing the likelihood of clinical adoption.

Statistical inference for net benefit measures in biomarker validation studies.

Referral strategies based on risk scores and medical tests are commonly proposed. Direct assessment of their clinical utility requires implementing the strategy and is not possible in the early phases of biomarker research. Prior to late-phase studies, net benefit measures can be used to assess the potential clinical impact of a proposed strategy. Validation studies, in which the biomarker defines a prespecified referral strategy, are a gold standard approach to evaluating biomarker potential. Uncertainty, quantified by a confidence interval, is important to consider when deciding whether a biomarker warrants an impact study, does not demonstrate clinical potential, or that more data are needed. We establish distribution theory for empirical estimators of net benefit and propose empirical estimators of variance. The primary results are for the most commonly employed estimators of net benefit: from cohort and unmatched case-control samples, and for point estimates and net benefit curves. Novel estimators of net benefit under stratified two-phase and categorically matched case-control sampling are proposed and distribution theory developed. Results for common variants of net benefit and for estimation from right-censored outcomes are also presented. We motivate and demonstrate the methodology with examples from lung cancer research and highlight its application to study design.

Pathologist characteristics associated with accuracy and reproducibility of melanocytic skin lesion interpretation.

BACKGROUND: Diagnostic interpretations of melanocytic skin lesions vary widely among pathologists, yet the underlying reasons remain unclear. OBJECTIVE: Identify pathologist characteristics associated with rates of accuracy and reproducibility. METHODS: Pathologists independently interpreted the same set of biopsy specimens from melanocytic lesions on 2 occasions. Diagnoses were categorized into 1 of 5 classes according to the Melanocytic Pathology Assessment Tool and Hierarchy for Diagnosis system. Reproducibility was determined by pathologists' concordance of diagnoses across 2 occasions. Accuracy was defined by concordance with a consensus reference standard. Associations of pathologist characteristics with reproducibility and accuracy were assessed individually and in multivariable logistic regression models. RESULTS: Rates of diagnostic reproducibility and accuracy were highest among pathologists with board certification and/or fellowship training in dermatopathology and in those with 5 or more years of experience. In addition, accuracy was high among pathologists with a higher proportion of melanocytic lesions in their caseload composition and higher volume of melanocytic lesions. LIMITATIONS: Data gathered in a test set situation by using a classification tool not currently in clinical use. CONCLUSION: Diagnoses are more accurate among pathologists with specialty training and those with more experience interpreting melanocytic lesions. These findings support the practice of referring difficult cases to more experienced pathologists to improve diagnostic accuracy, although the impact of these referrals on patient outcomes requires additional research.

Diagnostic Reproducibility: What Happens When the Same Pathologist Interprets the Same Breast Biopsy Specimen at Two Points in Time?

BACKGROUND: Surgeons may receive a different diagnosis when a breast biopsy is interpreted by a second pathologist. The extent to which diagnostic agreement by the same pathologist varies at two time points is unknown. METHODS: Pathologists from eight U.S. states independently interpreted 60 breast specimens, one glass slide per case, on two occasions separated by ≥9 months. Reproducibility was assessed by comparing interpretations between the two time points; associations between reproducibility (intraobserver agreement rates); and characteristics of pathologists and cases were determined and also compared with interobserver agreement of baseline interpretations. RESULTS: Sixty-five percent of invited, responding pathologists were eligible and consented; 49 interpreted glass slides in both study phases, resulting in 2940 interpretations. Intraobserver agreement rates between the two phases were 92% [95% confidence interval (CI) 88-95] for invasive breast cancer, 84% (95% CI 81-87) for ductal carcinoma-in-situ, 53% (95% CI 47-59) for atypia, and 84% (95% CI 81-86) for benign without atypia. When comparing all study participants' case interpretations at baseline, interobserver agreement rates were 89% (95% CI 84-92) for invasive cancer, 79% (95% CI 76-81) for ductal carcinoma-in-situ, 43% (95% CI 41-45) for atypia, and 77% (95% CI 74-79) for benign without atypia. CONCLUSIONS: Interpretive agreement between two time points by the same individual pathologist was low for atypia and was similar to observed rates of agreement for atypia between different pathologists. Physicians and patients should be aware of the diagnostic challenges associated with a breast biopsy diagnosis of atypia when considering treatment and surveillance decisions.

Variability in Pathologists' Interpretations of Individual Breast Biopsy Slides: A Population Perspective.

BACKGROUND: The effect of physician diagnostic variability on accuracy at a population level depends on the prevalence of diagnoses. OBJECTIVE: To estimate how diagnostic variability affects accuracy from the perspective of a U.S. woman aged 50 to 59 years having a breast biopsy. DESIGN: Applied probability using Bayes' theorem. SETTING: B-Path (Breast Pathology) Study comparing pathologists' interpretations of a single biopsy slide versus a reference consensus interpretation from 3 experts. PARTICIPANTS: 115 practicing pathologists (6900 total interpretations from 240 distinct cases). MEASUREMENTS: A single representative slide from each of the 240 cases was used to estimate the proportion of biopsies with a diagnosis that would be verified if the same slide were interpreted by a reference group of 3 expert pathologists. Probabilities of confirmation (predictive values) were estimated using B-Path Study results and prevalence of biopsy diagnoses for women aged 50 to 59 years in the Breast Cancer Surveillance Consortium. RESULTS: Overall, if 1 representative slide were used per case, 92.3% (95% CI, 91.4% to 93.1%) of breast biopsy diagnoses would be verified by reference consensus diagnoses, with 4.6% (CI, 3.9% to 5.3%) overinterpreted and 3.2% (CI, 2.7% to 3.6%) underinterpreted. Verification of invasive breast cancer and benign without atypia diagnoses is highly probable; estimated predictive values were 97.7% (CI, 96.5% to 98.7%) and 97.1% (CI, 96.7% to 97.4%), respectively. Verification is less probable for atypia (53.6% overinterpreted and 8.6% underinterpreted) and ductal carcinoma in situ (DCIS) (18.5% overinterpreted and 11.8% underinterpreted). LIMITATIONS: Estimates are based on a testing situation with 1 slide used per case and without access to second opinions. Population-adjusted estimates may differ for women from other age groups, unscreened women, or women in different practice settings. CONCLUSION: This analysis, based on interpretation of a single breast biopsy slide per case, predicts a low likelihood that a diagnosis of atypia or DCIS would be verified by a reference consensus diagnosis. This diagnostic grey zone should be considered in clinical management decisions in patients with these diagnoses. PRIMARY FUNDING SOURCE: National Cancer Institute.

Early-Phase Studies of Biomarkers: What Target Sensitivity and Specificity Values Might Confer Clinical Utility?

BACKGROUND: Many cancer biomarker research studies seek to develop markers that can accurately detect or predict future onset of disease. To design and evaluate these studies, one must specify the levels of accuracy sought. However, justified target levels are rarely available. METHODS: We describe a way to calculate target levels of sensitivity and specificity for a biomarker intended to be applied in a defined clinical context. The calculation requires knowledge of the prevalence or incidence of cases in the clinical population and the ratio of benefit associated with the clinical consequences of a positive biomarker test in cases (true positive) to cost associated with a positive biomarker test in controls (false positive). Guidance is offered on soliciting the cost/benefit ratio. The calculations are based on the longstanding decision theory concept of providing a net benefit on average in the population, and they rely on some assumptions about uniformity of costs and benefits to those tested. RESULTS: Calculations are illustrated with 3 applications: predicting colon cancer recurrence in stage 1 patients; predicting interval breast cancer (between mammography screenings); and screening for ovarian cancer. CONCLUSIONS: It is feasible to specify target levels of biomarker performance that enable evaluation of the potential clinical impact of biomarkers in early-phase studies. Nevertheless, biomarkers meeting the criteria should still be tested rigorously in studies that measure the actual impact on patient outcomes of using the biomarker to make clinical decisions.

Designing a study to evaluate the benefit of a biomarker for selecting patient treatment.

Biomarkers that predict the efficacy of treatment can potentially improve clinical outcomes and decrease medical costs by allowing treatment to be provided only to those most likely to benefit. We consider the design of a randomized clinical trial in which one objective is to evaluate a treatment selection marker. The marker may be measured prospectively or retrospectively using samples collected at baseline. We describe and contrast criteria around which the trial can be designed. An existing approach focuses on determining if there is a statistical interaction between the marker and treatment. We propose three alternative approaches based on estimating clinically relevant measures of improvement in outcomes with use of the marker. Importantly, our approaches accommodate the common scenario in which the marker-based rule for recommending treatment is developed with data from the trial. Sample sizes are calculated for powering a trial to assess these criteria in the context of adjuvant chemotherapy for the treatment of estrogen-receptor-positive, node-positive breast cancer. In this example, we find that larger sample sizes are generally required for assessing clinical impact than for simply evaluating if there is a statistical interaction between marker and treatment. We also find that retrospectively selecting a case-control subset of subjects for marker evaluation can lead to large efficiency gains, especially if cases and controls are matched on treatment assignment.

Diagnostic concordance among pathologists interpreting breast biopsy specimens.

IMPORTANCE: A breast pathology diagnosis provides the basis for clinical treatment and management decisions; however, its accuracy is inadequately understood. OBJECTIVES: To quantify the magnitude of diagnostic disagreement among pathologists compared with a consensus panel reference diagnosis and to evaluate associated patient and pathologist characteristics. DESIGN, SETTING, AND PARTICIPANTS: Study of pathologists who interpret breast biopsies in clinical practices in 8 US states. EXPOSURES: Participants independently interpreted slides between November 2011 and May 2014 from test sets of 60 breast biopsies (240 total cases, 1 slide per case), including 23 cases of invasive breast cancer, 73 ductal carcinoma in situ (DCIS), 72 with atypical hyperplasia (atypia), and 72 benign cases without atypia. Participants were blinded to the interpretations of other study pathologists and consensus panel members. Among the 3 consensus panel members, unanimous agreement of their independent diagnoses was 75%, and concordance with the consensus-derived reference diagnoses was 90.3%. MAIN OUTCOMES AND MEASURES: The proportions of diagnoses overinterpreted and underinterpreted relative to the consensus-derived reference diagnoses were assessed. RESULTS: Sixty-five percent of invited, responding pathologists were eligible and consented to participate. Of these, 91% (N = 115) completed the study, providing 6900 individual case diagnoses. Compared with the consensus-derived reference diagnosis, the overall concordance rate of diagnostic interpretations of participating pathologists was 75.3% (95% CI, 73.4%-77.0%; 5194 of 6900 interpretations). Among invasive carcinoma cases (663 interpretations), 96% (95% CI, 94%-97%) were concordant, and 4% (95% CI, 3%-6%) were underinterpreted; among DCIS cases (2097 interpretations), 84% (95% CI, 82%-86%) were concordant, 3% (95% CI, 2%-4%) were overinterpreted, and 13% (95% CI, 12%-15%) were underinterpreted; among atypia cases (2070 interpretations), 48% (95% CI, 44%-52%) were concordant, 17% (95% CI, 15%-21%) were overinterpreted, and 35% (95% CI, 31%-39%) were underinterpreted; and among benign cases without atypia (2070 interpretations), 87% (95% CI, 85%-89%) were concordant and 13% (95% CI, 11%-15%) were overinterpreted. Disagreement with the reference diagnosis was statistically significantly higher among biopsies from women with higher (n = 122) vs lower (n = 118) breast density on prior mammograms (overall concordance rate, 73% [95% CI, 71%-75%] for higher vs 77% [95% CI, 75%-80%] for lower, P < .001), and among pathologists who interpreted lower weekly case volumes (P < .001) or worked in smaller practices (P = .034) or nonacademic settings (P = .007). CONCLUSIONS AND RELEVANCE: In this study of pathologists, in which diagnostic interpretation was based on a single breast biopsy slide, overall agreement between the individual pathologists' interpretations and the expert consensus-derived reference diagnoses was 75.3%, with the highest level of concordance for invasive carcinoma and lower levels of concordance for DCIS and atypia. Further research is needed to understand the relationship of these findings with patient management.

Net reclassification indices for evaluating risk prediction instruments: a critical review.

Net reclassification indices have recently become popular statistics for measuring the prediction increment of new biomarkers. We review the various types of net reclassification indices and their correct interpretations. We evaluate the advantages and disadvantages of quantifying the prediction increment with these indices. For predefined risk categories, we relate net reclassification indices to existing measures of the prediction increment. We also consider statistical methodology for constructing confidence intervals for net reclassification indices and evaluate the merits of hypothesis testing based on such indices. We recommend that investigators using net reclassification indices should report them separately for events (cases) and nonevents (controls). When there are two risk categories, the components of net reclassification indices are the same as the changes in the true- and false-positive rates. We advocate the use of true- and false-positive rates and suggest it is more useful for investigators to retain the existing, descriptive terms. When there are three or more risk categories, we recommend against net reclassification indices because they do not adequately account for clinically important differences in shifts among risk categories. The category-free net reclassification index is a new descriptive device designed to avoid predefined risk categories. However, it experiences many of the same problems as other measures such as the area under the receiver operating characteristic curve. In addition, the category-free index can mislead investigators by overstating the incremental value of a biomarker, even in independent validation data. When investigators want to test a null hypothesis of no prediction increment, the well-established tests for coefficients in the regression model are superior to the net reclassification index. If investigators want to use net reclassification indices, confidence intervals should be calculated using bootstrap methods rather than published variance formulas. The preferred single-number summary of the prediction increment is the improvement in net benefit.

Adopting nested case-control quota sampling designs for the evaluation of risk markers.

Two-phase study methods, in which more detailed or more expensive exposure information is only collected on a sample of individuals with events and a small proportion of other individuals, are expected to play a critical role in biomarker validation research. One major limitation of standard two-phase designs is that they are most conveniently employed with study cohorts in which information on longitudinal follow-up and other potential matching variables is electronically recorded. However for many practical situations, at the sampling stage such information may not be readily available for every potential candidates. Study eligibility needs to be verified by reviewing information from medical charts one by one. In this manuscript, we study in depth a novel study design commonly undertaken in practice that involves sampling until quotas of eligible cases and controls are identified. We propose semiparametric methods to calculate risk distributions and a wide variety of prediction indices when outcomes are censored failure times and data are collected under the quota sampling design. Consistency and asymptotic normality of our estimators are established using empirical process theory. Simulation results indicate that the proposed procedures perform well in finite samples. Application is made to the evaluation of a new risk model for predicting the onset of cardiovascular disease.

Further insight into the incremental value of new markers: the interpretation of performance measures and the importance of clinical context.

In this issue of the Journal, Pencina and et al. (Am J Epidemiol. 2012;176(6):492-494) examine the operating characteristics of measures of incremental value. Their goal is to provide benchmarks for the measures that can help identify the most promising markers among multiple candidates. They consider a setting in which new predictors are conditionally independent of established predictors. In the present article, the authors consider more general settings. Their results indicate that some of the conclusions made by Pencina et al. are limited to the specific scenarios the authors considered. For example, Pencina et al. observed that continuous net reclassification improvement was invariant to the strength of the baseline model, but the authors of the present study show this invariance does not hold generally. Further, they disagree with the suggestion that such invariance would be desirable for a measure of incremental value. They also do not see evidence to support the claim that the measures provide complementary information. In addition, they show that correlation with baseline predictors can lead to much bigger gains in performance than the conditional independence scenario studied by Pencina et al. Finally, the authors note that the motivation of providing benchmarks actually reinforces previous observations that the problem with these measures is they do not have useful clinical interpretations. If they did, researchers could use the measures directly and benchmarks would not be needed.

Measuring the performance of markers for guiding treatment decisions.

Treatment selection markers, sometimes called predictive markers, are factors that help clinicians select therapies that maximize good outcomes and minimize adverse outcomes for patients. Existing statistical methods for evaluating a treatment selection marker include assessing its prognostic value, evaluating treatment effects in patients with a restricted range of marker values, and testing for a statistical interaction between marker value and treatment. These methods are inadequate, because they give misleading measures of performance that do not answer key clinical questions about how the marker might help patients choose treatment, how treatment decisions should be made on the basis of a continuous marker measurement, what effect using the marker to select treatment would have on the population, or what proportion of patients would have treatment changes on the basis of marker measurement. Marker-by-treatment predictiveness curves are proposed as a more useful aid to answering these clinically relevant questions, because they illustrate treatment effects as a function of marker value, outcomes when using or not using the marker to select treatment, and the proportion of patients for whom treatment recommendations change after marker measurement. Randomized therapeutic clinical trials, in which entry criteria and treatment regimens are not restricted by the marker, are also proposed as the basis for constructing the curves and evaluating and comparing markers.

Problems with risk reclassification methods for evaluating prediction models.

For comparing the performance of a baseline risk prediction model with one that includes an additional predictor, a risk reclassification analysis strategy has been proposed. The first step is to cross-classify risks calculated according to the 2 models for all study subjects. Summary measures including the percentage of reclassification and the percentage of correct reclassification are calculated, along with 2 reclassification calibration statistics. The author shows that interpretations of the proposed summary measures and P values are problematic. The author's recommendation is to display the reclassification table, because it shows interesting information, but to use alternative methods for summarizing and comparing model performance. The Net Reclassification Index has been suggested as one alternative method. The author argues for reporting components of the Net Reclassification Index because they are more clinically relevant than is the single numerical summary measure.

Joint modeling, covariate adjustment, and interaction: contrasting notions in risk prediction models and risk prediction performance.

Epidemiologic methods are well established for investigating the association of a predictor of interest and disease status in the presence of covariates also associated with disease. There is less consensus on how to handle covariates when the goal is to evaluate the increment in prediction performance gained by a new marker when a set of predictors already exists. We distinguish between adjusting for covariates and joint modeling of disease risk in this context. We show that adjustment and joint modeling are distinct concepts, and we describe the specific conditions where they are the same. We also discuss the concept of interaction among variables and describe a notion of interaction that is relevant to prediction performance. We conclude with a discussion of the most appropriate methods for evaluating new biomarkers in the presence of existing predictors.

Adding Rigor to Biomarker Evaluations-EDRN Experience.

The cancer early-detection biomarker field was, compared with the therapeutic arena, in its infancy when the Early Detection Research Network (EDRN) was initiated in 2000. The EDRN has played a crucial role in changing the culture and the ways people conduct biomarker studies. The EDRN proposed biomarker developmental guidelines and biomarker pivotal trial study design standards, created biomarker reference sets and functioned as an unbiased broker for the field, implemented the most rigorous blinding policy in the biomarker field, developed an array of statistical and computational tools for early-detection biomarker evaluations, and developed a multidisciplinary team-science approach. We reviewed these contributions made by the EDRN and their impacts on maturing the field. Future challenges and opportunities in cancer early-detection biomarker translational research are discussed, particularly in strengthening biomarker discovery pipeline and conducting more efficient biomarker validation studies.See all articles in this CEBP Focus section, "NCI Early Detection Research Network: Making Cancer Detection Possible."

Assessing the value of risk predictions by using risk stratification tables.

The recent epidemiologic and clinical literature is filled with studies evaluating statistical models for predicting disease or some other adverse event. Risk stratification tables are a new way to evaluate the benefit of adding a new risk marker to a risk prediction model that includes an established set of markers. This approach involves cross-tabulating risk predictions from models with and without the new marker. In this article, the authors use examples to show how risk stratification tables can be used to compare 3 important measures of model performance between the models with and those without the new marker: the extent to which the risks calculated from the models reflect the actual fraction of persons in the population with events (calibration); the proportions in which the population is stratified into clinically relevant risk categories (stratification capacity); and the extent to which participants with events are assigned to high-risk categories and those without events are assigned to low-risk categories (classification accuracy). They detail common misinterpretations and misuses of the risk stratification method and conclude that the information that can be extracted from risk stratification tables is an enormous improvement over commonly reported measures of risk prediction model performance (for example, c-statistics and Hosmer-Lemeshow tests) because it describes the value of the models for guiding medical decisions.

Assessment of Second-Opinion Strategies for Diagnoses of Cutaneous Melanocytic Lesions.

Importance: Histopathologic criteria have limited diagnostic reliability for a range of cutaneous melanocytic lesions. Objective: To evaluate the association of second-opinion strategies by general pathologists and dermatopathologists with the overall reliability of diagnosis of difficult melanocytic lesions. Design, Setting, and Participants: This diagnostic study used samples from the Melanoma Pathology Study, which comprises 240 melanocytic lesion samples selected from a dermatopathology laboratory in Bellevue, Washington, and represents the full spectrum of lesions from common nevi to invasive melanoma. Five sets of 48 samples were evaluated independently by 187 US pathologists from July 15, 2013, through May 23, 2016. Data analysis was performed from April 2016 through November 2017. Main Outcomes and Measures: Accuracy of diagnosis, defined as concordance with an expert consensus diagnosis of 3 experienced pathologists, was assessed after applying 10 different second-opinion strategies. Results: Among the 187 US pathologists examining the 24 lesion samples, 113 were general pathologists (65 men [57.5%]; mean age at survey, 53.7 years [range, 33.0-79.0 years]) and 74 were dermatopathologists (49 men [66.2%]; mean age at survey, 46.4 years [range, 33.0-77.0 years]). Among the 8976 initial case interpretations, physicians desired second opinions for 3899 (43.4%), most often for interpretation of severely dysplastic nevi. The overall misclassification rate was highest when interpretations did not include second opinions and initial reviewers were all general pathologists lacking subspecialty training (52.8%; 95% CI, 51.3%-54.3%). When considering different second opinion strategies, the misclassification of melanocytic lesions was lowest when the first, second, and third consulting reviewers were subspecialty-trained dermatopathologists and when all lesions were subject to second opinions (36.7%; 95% CI, 33.1%-40.7%). When the second opinion strategies were compared with single interpretations without second opinions, the reductions in misclassification rates for some of the strategies were statistically significant, but none of the strategies eliminated diagnostic misclassification. Melanocytic lesions in the middle of the diagnostic spectrum had the highest misclassification rates (eg, moderately or severely dysplastic nevus, Spitz nevus, melanoma in situ, and pathologic stage [p]T1a invasive melanoma). Variability of in situ and thin invasive melanoma was relatively intractable to all examined strategies. Conclusions and Relevance: The results of this study suggest that second opinions rendered by dermatopathologists improve reliability of melanocytic lesion diagnosis. However, discordance among pathologists remained high.

Adjusting for covariates in studies of diagnostic, screening, or prognostic markers: an old concept in a new setting.

The concept of covariate adjustment is well established in therapeutic and etiologic studies. However, it has received little attention in the growing area of medical research devoted to the development of markers for disease diagnosis, screening, or prognosis, where classification accuracy, rather than association, is of primary interest. In this paper, the authors demonstrate the need for covariate adjustment in studies of classification accuracy, discuss methods for adjusting for covariates, and distinguish covariate adjustment from several other related, but fundamentally different, uses for covariates. They draw analogies and contrasts throughout with studies of association.