Internal Standard Triggered-Parallel Reaction Monitoring Mass Spectrometry Enables Multiplexed Quantification of Candidate Biomarkers in Plasma.

Despite advances in proteomic technologies, clinical translation of plasma biomarkers remains low, partly due to a major bottleneck between the discovery of candidate biomarkers and costly clinical validation studies. Due to a dearth of multiplexable assays, generally only a few candidate biomarkers are tested, and the validation success rate is accordingly low. Previously, mass spectrometry-based approaches have been used to fill this gap but feature poor quantitative performance and were generally limited to hundreds of proteins. Here, we demonstrate the capability of an internal standard triggered-parallel reaction monitoring (IS-PRM) assay to greatly expand the numbers of candidates that can be tested with improved quantitative performance. The assay couples immunodepletion and fractionation with IS-PRM and was developed and implemented in human plasma to quantify 5176 peptides representing 1314 breast cancer biomarker candidates. Characterization of the IS-PRM assay demonstrated the precision (median % CV of 7.7%), linearity (median R2 > 0.999 over 4 orders of magnitude), and sensitivity (median LLOQ < 1 fmol, approximately) to enable rank-ordering of candidate biomarkers for validation studies. Using three plasma pools from breast cancer patients and three control pools, 893 proteins were quantified, of which 162 candidate biomarkers were verified in at least one of the cancer pools and 22 were verified in all three cancer pools. The assay greatly expands capabilities for quantification of large numbers of proteins and is well suited for prioritization of viable candidate biomarkers.

Histopathological diagnosis of cutaneous melanocytic lesions: blinded and nonblinded second opinions offer similar improvement in diagnostic accuracy.

BACKGROUND: Previous studies of second opinions in the diagnosis of melanocytic skin lesions have examined blinded second opinions, which do not reflect usual clinical practice. The current study, conducted in the USA, investigated both blinded and nonblinded second opinions for their impact on diagnostic accuracy. METHODS: In total, 100 melanocytic skin biopsy cases, ranging from benign to invasive melanoma, were interpreted by 74 dermatopathologists. Subsequently, 151 dermatopathologists performed nonblinded second and third reviews. We compared the accuracy of single reviewers, second opinions obtained from independent, blinded reviewers and second opinions obtained from sequential, nonblinded reviewers. Accuracy was defined with respect to a consensus reference diagnosis. RESULTS: The mean case-level diagnostic accuracy of single reviewers was 65.3% (95% CI 63.4-67.2%). Second opinions arising from sequential, nonblinded reviewers significantly improved accuracy to 69.9% (95% CI 68.0-71.7%; P < 0.001). Similarly, second opinions arising from blinded reviewers improved upon the accuracy of single reviewers (69.2%; 95% CI 68.0-71.7%). Nonblinded reviewers were more likely than blinded reviewers to give diagnoses in the same diagnostic classes as the first diagnosis. Nonblinded reviewers tended to be more confident when they agreed with previous reviewers, even with inaccurate diagnoses. CONCLUSION: We found that both blinded and nonblinded second reviewers offered a similar modest improvement in diagnostic accuracy compared with single reviewers. Obtaining second opinions with knowledge of previous reviews tends to generate agreement among reviews, and may generate unwarranted confidence in an inaccurate diagnosis. Combining aspects of both blinded and nonblinded review in practice may leverage the advantages while mitigating the disadvantages of each approach. Specifically, a second pathologist could give an initial diagnosis blinded to the results of the first pathologist, with subsequent nonblinded discussion between the two pathologists if their diagnoses differ.

Use of the MyProstateScore Test to Rule Out Clinically Significant Cancer: Validation of a Straightforward Clinical Testing Approach.

PURPOSE: The MyProstateScore test was validated for improved detection of clinically significant (grade group ≥2) prostate cancer relative to prostate specific antigen based risk calculators. We sought to validate an optimal MyProstateScore threshold for clinical use in ruling out grade group ≥2 cancer in men referred for biopsy. MATERIALS AND METHODS: Biopsy naïve men provided post-digital rectal examination urine prior to biopsy. MyProstateScore was calculated using the validated, locked multivariable model including only serum prostate specific antigen, urinary prostate cancer antigen 3 and urinary TMPRSS2:ERG. The MyProstateScore threshold approximating 95% sensitivity for grade group ≥2 cancer was identified in a training cohort, and performance was measured in 2 external validation cohorts. We assessed the 1) overall biopsy referral population and 2) population meeting guideline based testing criteria (ie, prostate specific antigen 3-10, or <3 with suspicious digital rectal examination). RESULTS: Validation cohorts were prospectively enrolled from academic (977 patients, median prostate specific antigen 4.5, IQR 3.1-6.0) and community (548, median prostate specific antigen 4.9, IQR 3.7-6.8) settings. In the overall validation population (1,525 patients), 338 men (22%) had grade group ≥2 cancer on biopsy. The MyProstateScore threshold of 10 provided 97% sensitivity and 98% negative predictive value for grade group ≥2 cancer. MyProstateScore testing would have prevented 387 unnecessary biopsies (33%), while missing only 10 grade group ≥2 cancers (3.0%). In 1,242 patients meeting guideline based criteria, MyProstateScore ≤10 provided 96% sensitivity and 97% negative predictive value, and would have prevented 32% of unnecessary biopsies, missing 3.7% of grade group ≥2 cancers. CONCLUSIONS: In a large, clinically pertinent biopsy referral population, MyProstateScore ≤10 provided exceptional sensitivity and negative predictive value for ruling out grade group ≥2 cancer. This straightforward secondary testing approach would reduce the use of more costly and invasive procedures after screening with prostate specific antigen.

Second opinion strategies in breast pathology: a decision analysis addressing over-treatment, under-treatment, and care costs.

PURPOSE: To estimate the potential near-term population impact of alternative second opinion breast biopsy pathology interpretation strategies. METHODS: Decision analysis examining 12-month outcomes of breast biopsy for nine breast pathology interpretation strategies in the U.S. health system. Diagnoses of 115 practicing pathologists in the Breast Pathology Study were compared to reference-standard-consensus diagnoses with and without second opinions. Interpretation strategies were defined by whether a second opinion was sought universally or selectively (e.g., 2nd opinion if invasive). Main outcomes were the expected proportion of concordant breast biopsy diagnoses, the proportion involving over- or under-interpretation, and cost of care in U.S. dollars within one-year of biopsy. RESULTS: Without a second opinion, 92.2% of biopsies received a concordant diagnosis. Concordance rates increased under all second opinion strategies, and the rate was highest (95.1%) and under-treatment lowest (2.6%) when all biopsies had second opinions. However, over-treatment was lowest when second opinions were sought selectively for initial diagnoses of invasive cancer, DCIS, or atypia (1.8 vs. 4.7% with no 2nd opinions). This strategy also had the lowest projected 12-month care costs ($5.907 billion vs. $6.049 billion with no 2nd opinions). CONCLUSIONS: Second opinion strategies could lower overall care costs while reducing both over- and under-treatment. The most accurate cost-saving strategy required second opinions for initial diagnoses of invasive cancer, DCIS, or atypia.

Pathologist characteristics associated with accuracy and reproducibility of melanocytic skin lesion interpretation.

BACKGROUND: Diagnostic interpretations of melanocytic skin lesions vary widely among pathologists, yet the underlying reasons remain unclear. OBJECTIVE: Identify pathologist characteristics associated with rates of accuracy and reproducibility. METHODS: Pathologists independently interpreted the same set of biopsy specimens from melanocytic lesions on 2 occasions. Diagnoses were categorized into 1 of 5 classes according to the Melanocytic Pathology Assessment Tool and Hierarchy for Diagnosis system. Reproducibility was determined by pathologists' concordance of diagnoses across 2 occasions. Accuracy was defined by concordance with a consensus reference standard. Associations of pathologist characteristics with reproducibility and accuracy were assessed individually and in multivariable logistic regression models. RESULTS: Rates of diagnostic reproducibility and accuracy were highest among pathologists with board certification and/or fellowship training in dermatopathology and in those with 5 or more years of experience. In addition, accuracy was high among pathologists with a higher proportion of melanocytic lesions in their caseload composition and higher volume of melanocytic lesions. LIMITATIONS: Data gathered in a test set situation by using a classification tool not currently in clinical use. CONCLUSION: Diagnoses are more accurate among pathologists with specialty training and those with more experience interpreting melanocytic lesions. These findings support the practice of referring difficult cases to more experienced pathologists to improve diagnostic accuracy, although the impact of these referrals on patient outcomes requires additional research.

Variability in Pathologists' Interpretations of Individual Breast Biopsy Slides: A Population Perspective.

BACKGROUND: The effect of physician diagnostic variability on accuracy at a population level depends on the prevalence of diagnoses. OBJECTIVE: To estimate how diagnostic variability affects accuracy from the perspective of a U.S. woman aged 50 to 59 years having a breast biopsy. DESIGN: Applied probability using Bayes' theorem. SETTING: B-Path (Breast Pathology) Study comparing pathologists' interpretations of a single biopsy slide versus a reference consensus interpretation from 3 experts. PARTICIPANTS: 115 practicing pathologists (6900 total interpretations from 240 distinct cases). MEASUREMENTS: A single representative slide from each of the 240 cases was used to estimate the proportion of biopsies with a diagnosis that would be verified if the same slide were interpreted by a reference group of 3 expert pathologists. Probabilities of confirmation (predictive values) were estimated using B-Path Study results and prevalence of biopsy diagnoses for women aged 50 to 59 years in the Breast Cancer Surveillance Consortium. RESULTS: Overall, if 1 representative slide were used per case, 92.3% (95% CI, 91.4% to 93.1%) of breast biopsy diagnoses would be verified by reference consensus diagnoses, with 4.6% (CI, 3.9% to 5.3%) overinterpreted and 3.2% (CI, 2.7% to 3.6%) underinterpreted. Verification of invasive breast cancer and benign without atypia diagnoses is highly probable; estimated predictive values were 97.7% (CI, 96.5% to 98.7%) and 97.1% (CI, 96.7% to 97.4%), respectively. Verification is less probable for atypia (53.6% overinterpreted and 8.6% underinterpreted) and ductal carcinoma in situ (DCIS) (18.5% overinterpreted and 11.8% underinterpreted). LIMITATIONS: Estimates are based on a testing situation with 1 slide used per case and without access to second opinions. Population-adjusted estimates may differ for women from other age groups, unscreened women, or women in different practice settings. CONCLUSION: This analysis, based on interpretation of a single breast biopsy slide per case, predicts a low likelihood that a diagnosis of atypia or DCIS would be verified by a reference consensus diagnosis. This diagnostic grey zone should be considered in clinical management decisions in patients with these diagnoses. PRIMARY FUNDING SOURCE: National Cancer Institute.

Achieving consensus for the histopathologic diagnosis of melanocytic lesions: use of the modified Delphi method.

OBJECTIVE: To understand the sophisticated nature of coming to consensus when diagnosing complex melanocytic lesions among a panel of experienced dermatopathologists. METHODS: A total of 240 melanocytic lesions were assessed independently by three experienced dermatopathologists with their diagnoses mapped into one of five Melanocytic Pathology Assessment Tool and Hierarchy for Diagnosis (MPATH-DX) categories: (I) nevus/mild atypia, (II) moderate atypia, (III) severe atypia/melanoma in situ, (IV) T1a invasive melanoma and (V) ≥ T1b invasive melanoma. The dermatopathologists then discussed the cases, using a modified Delphi method to facilitated consensus building for cases with discordant diagnoses. RESULTS: For most cases, a majority of interpretations (two or three of three) agreed with the consensus diagnosis in 95% of Category I, 64% of Category II, 84% of Category III, 88% for Category IV and 100% of Category V cases. Disagreements were typically due to diagnostic threshold differences (64.5%), differing contents on slides even though the slides were sequential cuts (18.5%), and missed findings (15.3%). Disagreements were resolved via discussion of histopathologic features and their significance while reviewing the slides using a multi-headed microscope, considering treatment recommendations, citing existing literature, reviewing additional slides for a case, and choosing a provisional/borderline diagnosis to capture diverse opinions. All experienced pathologists participating in this study reported that the process of coming to consensus was challenging for borderline cases and may have represented compromise rather than consensus. They also reported the process changed their approaches to diagnosing complex melanocytic lesions. CONCLUSIONS: The most frequent reason for disagreement of experienced dermatopathologists was differences in diagnostic thresholds related to observer viewpoints. A range of approaches was needed to come to consensus, and this may guide pathology groups who do not currently hold consensus conferences.

Diagnostic concordance among pathologists interpreting breast biopsy specimens.

IMPORTANCE: A breast pathology diagnosis provides the basis for clinical treatment and management decisions; however, its accuracy is inadequately understood. OBJECTIVES: To quantify the magnitude of diagnostic disagreement among pathologists compared with a consensus panel reference diagnosis and to evaluate associated patient and pathologist characteristics. DESIGN, SETTING, AND PARTICIPANTS: Study of pathologists who interpret breast biopsies in clinical practices in 8 US states. EXPOSURES: Participants independently interpreted slides between November 2011 and May 2014 from test sets of 60 breast biopsies (240 total cases, 1 slide per case), including 23 cases of invasive breast cancer, 73 ductal carcinoma in situ (DCIS), 72 with atypical hyperplasia (atypia), and 72 benign cases without atypia. Participants were blinded to the interpretations of other study pathologists and consensus panel members. Among the 3 consensus panel members, unanimous agreement of their independent diagnoses was 75%, and concordance with the consensus-derived reference diagnoses was 90.3%. MAIN OUTCOMES AND MEASURES: The proportions of diagnoses overinterpreted and underinterpreted relative to the consensus-derived reference diagnoses were assessed. RESULTS: Sixty-five percent of invited, responding pathologists were eligible and consented to participate. Of these, 91% (N = 115) completed the study, providing 6900 individual case diagnoses. Compared with the consensus-derived reference diagnosis, the overall concordance rate of diagnostic interpretations of participating pathologists was 75.3% (95% CI, 73.4%-77.0%; 5194 of 6900 interpretations). Among invasive carcinoma cases (663 interpretations), 96% (95% CI, 94%-97%) were concordant, and 4% (95% CI, 3%-6%) were underinterpreted; among DCIS cases (2097 interpretations), 84% (95% CI, 82%-86%) were concordant, 3% (95% CI, 2%-4%) were overinterpreted, and 13% (95% CI, 12%-15%) were underinterpreted; among atypia cases (2070 interpretations), 48% (95% CI, 44%-52%) were concordant, 17% (95% CI, 15%-21%) were overinterpreted, and 35% (95% CI, 31%-39%) were underinterpreted; and among benign cases without atypia (2070 interpretations), 87% (95% CI, 85%-89%) were concordant and 13% (95% CI, 11%-15%) were overinterpreted. Disagreement with the reference diagnosis was statistically significantly higher among biopsies from women with higher (n = 122) vs lower (n = 118) breast density on prior mammograms (overall concordance rate, 73% [95% CI, 71%-75%] for higher vs 77% [95% CI, 75%-80%] for lower, P < .001), and among pathologists who interpreted lower weekly case volumes (P < .001) or worked in smaller practices (P = .034) or nonacademic settings (P = .007). CONCLUSIONS AND RELEVANCE: In this study of pathologists, in which diagnostic interpretation was based on a single breast biopsy slide, overall agreement between the individual pathologists' interpretations and the expert consensus-derived reference diagnoses was 75.3%, with the highest level of concordance for invasive carcinoma and lower levels of concordance for DCIS and atypia. Further research is needed to understand the relationship of these findings with patient management.

Testing for improvement in prediction model performance.

Authors have proposed new methodology in recent years for evaluating the improvement in prediction performance gained by adding a new predictor, Y, to a risk model containing a set of baseline predictors, X, for a binary outcome D. We prove theoretically that null hypotheses concerning no improvement in performance are equivalent to the simple null hypothesis that Y is not a risk factor when controlling for X, H0 : P(D = 1 | X,Y ) = P(D = 1 | X). Therefore, testing for improvement in prediction performance is redundant if Y has already been shown to be a risk factor. We also investigate properties of tests through simulation studies, focusing on the change in the area under the ROC curve (AUC). An unexpected finding is that standard testing procedures that do not adjust for variability in estimated regression coefficients are extremely conservative. This may explain why the AUC is widely considered insensitive to improvements in prediction performance and suggests that the problem of insensitivity has to do with use of invalid procedures for inference rather than with the measure itself. To avoid redundant testing and use of potentially problematic methods for inference, we recommend that hypothesis testing for no improvement be limited to evaluation of Y as a risk factor, for which methods are well developed and widely available. Analyses of measures of prediction performance should focus on estimation rather than on testing for no improvement in performance.

Development of a diagnostic test set to assess agreement in breast pathology: practical application of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS).

BACKGROUND: Diagnostic test sets are a valuable research tool that contributes importantly to the validity and reliability of studies that assess agreement in breast pathology. In order to fully understand the strengths and weaknesses of any agreement and reliability study, however, the methods should be fully reported. In this paper we provide a step-by-step description of the methods used to create four complex test sets for a study of diagnostic agreement among pathologists interpreting breast biopsy specimens. We use the newly developed Guidelines for Reporting Reliability and Agreement Studies (GRRAS) as a basis to report these methods. METHODS: Breast tissue biopsies were selected from the National Cancer Institute-funded Breast Cancer Surveillance Consortium sites. We used a random sampling stratified according to woman's age (40-49 vs. ≥50), parenchymal breast density (low vs. high) and interpretation of the original pathologist. A 3-member panel of expert breast pathologists first independently interpreted each case using five primary diagnostic categories (non-proliferative changes, proliferative changes without atypia, atypical ductal hyperplasia, ductal carcinoma in situ, and invasive carcinoma). When the experts did not unanimously agree on a case diagnosis a modified Delphi method was used to determine the reference standard consensus diagnosis. The final test cases were stratified and randomly assigned into one of four unique test sets. CONCLUSIONS: We found GRRAS recommendations to be very useful in reporting diagnostic test set development and recommend inclusion of two additional criteria: 1) characterizing the study population and 2) describing the methods for reference diagnosis, when applicable.

Pathway-specific tumor suppression. Reduction of p27 accelerates gastrointestinal tumorigenesis in Apc mutant mice, but not in Smad3 mutant mice.

Expression of the cyclin-dependent kinase inhibitor p27(Kip1) (p27) is frequently reduced in human colorectal cancer, and this correlates with poor patient prognosis. To clarify the role of p27 in gastrointestinal (GI) cancer, we measured p27 expression, as well as the effect of germline deletion of p27, in 3 different mouse models of GI neoplasia. p27 expression was frequently reduced in GI tumors arising in 1,2-dimethylhydrazine (DMH) treated mice, and in Apc mutant Min/+ mice, but not in GI tumors arising in Smad3 mutant mice. Germline deletion of p27 resulted in accelerated tumor development and increased tumor cell proliferation in both DMH treated and Min/+ mice, but not in Smad3 mutant mice. p27 deficiency also led to increased adenoma to adenocarcinoma progression. These results indicate that reduction of p27 cooperates with mutations in Apc but not in Smad3 during GI tumorigenesis. Thus, tumor suppression by p27 is contingent on the specific oncogenic pathway that drives tumor development.

TRAIL-R deficiency in mice enhances lymph node metastasis without affecting primary tumor development.

TRAIL is a promising anticancer agent due to its ability to selectively induce apoptosis in established tumor cell lines but not nontransformed cells. Herein, we demonstrate a role for the apoptosis-inducing TRAIL receptor (TRAIL-R) as a metastasis suppressor. Although mouse models employing tumor transplantation have shown that TRAIL can reduce tumor growth, autochthonous tumor models have generated conflicting results with respect to the physiological role of the TRAIL system during tumorigenesis. We used a multistage model of squamous cell carcinoma to examine the role of TRAIL-R throughout all steps of tumor development. DMBA/TPA-treated TRAIL-R-deficient mice showed neither an increase in number or growth rate of benign papillomas nor an increase in the rate of progression to squamous cell carcinoma. However, metastasis to lymph nodes was significantly enhanced, indicating a role for TRAIL-R specifically in the suppression of metastasis. We also found that adherent TRAIL-R-expressing skin carcinoma cells were TRAIL resistant in vitro but were sensitized to TRAIL upon detachment by inactivation of the ERK signaling pathway. As detachment from the primary tumor is an obligatory step in metastasis, this provides a possible mechanism by which TRAIL-R could inhibit metastasis. Hence, treatment of cancer patients with agonists of the apoptosis-inducing receptors for TRAIL may prove useful in reducing the incidence of metastasis.

Plasma miRNA Biomarkers in Limited Volume Samples for Detection of Early-stage Pancreatic Cancer.

Early detection of pancreatic ductal adenocarcinoma (PDAC) is key to improving patient outcomes; however, PDAC is usually diagnosed late. Therefore, blood-based minimally invasive biomarker assays for limited volume clinical samples are urgently needed. A novel miRNA profiling platform (Abcam Fireplex-Oncology Panel) was used to investigate the feasibility of developing early detection miRNA biomarkers with 20 µL plasma from a training set (58 stage II PDAC cases and 30 controls) and two validation sets (34 stage II PDAC cases and 25 controls; 44 stage II PDAC cases and 18 controls). miR-34a-5p [AUC = 0.77; 95% confidence interval (CI), 0.66-0.87], miR-130a-3p (AUC = 0.74; 95% CI, 0.63-0.84), and miR-222-3p (AUC = 0.70; 95% CI, 0.58-0.81) were identified as significantly differentially abundant in plasma from stage II PDAC versus controls. Although none of the miRNAs individually outperformed the currently used serologic biomarker for PDAC, carbohydrate antigen 19-9 (CA19-9), combining the miRNAs with CA 19-9 improved AUCs from 0.89 (95% CI, 0.81-0.95) for CA 19-9 alone to 0.92 (95% CI, 0.86-0.97), 0.94 (95% CI, 0.89-0.98), and 0.92 (95% CI, 0.87-0.97), respectively. Gene set enrichment analyses of transcripts correlated with high and low expression of the three miRNAs in The Cancer Genome Atlas PDAC sample set. These miRNA biomarkers, assayed in limited volume plasma together with CA19-9, discriminate stage II PDAC from controls with good sensitivity and specificity. Unbiased profiling of larger cohorts should help develop an informative early detection biomarker assay for diagnostic settings. PREVENTION RELEVANCE: Development of minimally invasive biomarker assays for detection of premalignant disease and early-stage pancreatic cancer is key to improving patient survival. This study describes a limited volume plasma miRNA biomarker assay that can detect early-stage resectable pancreatic cancer in clinical samples necessary for effective prevention and clinical intervention.

Association of Second-Opinion Strategies in the Histopathologic Diagnosis of Cutaneous Melanocytic Lesions With Diagnostic Accuracy and Population-Level Costs.

IMPORTANCE: Diagnostic variation among pathologists interpreting cutaneous melanocytic lesions could lead to suboptimal care. OBJECTIVE: To estimate the potential association of second-opinion strategies in the histopathologic diagnosis of cutaneous melanocytic lesions with diagnostic accuracy and 1-year population-level costs in the US. DESIGN, SETTING, AND PARTICIPANTS: Decision analysis with 1-year time horizon including melanocytic lesion diagnoses available from US pathologists participating in the Melanoma Pathology Study (M-Path) and from the study panel of reference pathologists who classified cases using the MPATH-Dx classification tool. M-Path data collection occurred from July 2013 through March 2015; analyses for the present study were performed between April 2015 and January 2021. EXPOSURES: Various second-opinion strategies for interpretation of melanocytic cutaneous lesions. MAIN OUTCOMES AND MEASURES: Estimated accuracy of pathologists' diagnoses, defined as concordance with the reference panel diagnoses, and 1-year postbiopsy medical costs under various second-opinion strategies. Expected percentage of concordant diagnoses, including percentages of overinterpretation and underinterpretation, and 1-year costs of medical care per 100 000 in the US population. RESULTS: Decision-analytic model parameters were based on diagnostic interpretations for 240 cases by 187 pathologists compared with reference panel diagnoses. Without second opinions, 83.2% of diagnoses in the US were estimated to be accurate-ie, concordant with the reference diagnosis; with overinterpretation (8.0%) or underinterpretation (8.8%), and 16 850 misclassified diagnoses per 100 000 biopsies. Accuracy increased under all second-opinion strategies. Accuracy (87.4% concordance with 3.6% overinterpretation and 9.1% underinterpretation) and cost (an increase of more than $10 million per 100 000 biopsies per year) were highest when second opinions were universal (eg, performed on all biopsies), relative to no second opinions. A selective second-opinion strategy based on pathologists' desire or institutional requirements for a second opinion was most accurate (86.5% concordance; 4.4% overinterpretation; 9.1% underinterpretation) and would reduce costs by more than $1.9 million per 100 000 skin biopsies relative to no second opinions. Improvements in diagnostic accuracy with all second-opinion strategies were associated with reductions in overinterpretation but not underinterpretation. CONCLUSIONS AND RELEVANCE: In this decision-analytic model, selective second-opinion strategies for interpretation of melanocytic skin lesions showed the potential to improve diagnostic accuracy and decrease costs relative to no second opinions or universal second opinions.

Diagnostic markers for early detection of ovarian cancer.

PURPOSE: Early detection would significantly decrease the mortality rate of ovarian cancer. In this study, we characterize and validate the combination of six serum biomarkers that discriminate between disease-free and ovarian cancer patients with high efficiency. EXPERIMENTAL DESIGN: We analyzed 362 healthy controls and 156 newly diagnosed ovarian cancer patients. Concentrations of leptin, prolactin, osteopontin, insulin-like growth factor II, macrophage inhibitory factor, and CA-125 were determined using a multiplex, bead-based, immunoassay system. All six markers were evaluated in a training set (181 samples from the control group and 113 samples from OC patients) and a test set (181 sample control group and 43 ovarian cancer). RESULTS: Multiplex and ELISA exhibited the same pattern of expression for all the biomarkers. None of the biomarkers by themselves were good enough to differentiate healthy versus cancer cells. However, the combination of the six markers provided a better differentiation than CA-125. Four models with <2% classification error in training sets all had significant improvement (sensitivity 84%-98% at specificity 95%) over CA-125 (sensitivity 72% at specificity 95%) in the test set. The chosen model correctly classified 221 out of 224 specimens in the test set, with a classification accuracy of 98.7%. CONCLUSIONS: We describe the first blood biomarker test with a sensitivity of 95.3% and a specificity of 99.4% for the detection of ovarian cancer. Six markers provided a significant improvement over CA-125 alone for ovarian cancer detection. Validation was performed with a blinded cohort. This novel multiplex platform has the potential for efficient screening in patients who are at high risk for ovarian cancer.

Estimation and Comparison of Receiver Operating Characteristic Curves.

The receiver operating characteristic (ROC) curve displays the capacity of a marker or diagnostic test to discriminate between two groups of subjects, cases versus controls. We present a comprehensive suite of Stata commands for performing ROC analysis. Non-parametric, semiparametric and parametric estimators are calculated. Comparisons between curves are based on the area or partial area under the ROC curve. Alternatively pointwise comparisons between ROC curves or inverse ROC curves can be made. Options to adjust these analyses for covariates, and to perform ROC regression are described in a companion article. We use a unified framework by representing the ROC curve as the distribution of the marker in cases after standardizing it to the control reference distribution.

Accommodating Covariates in ROC Analysis.

Classification accuracy is the ability of a marker or diagnostic test to discriminate between two groups of individuals, cases and controls, and is commonly summarized using the receiver operating characteristic (ROC) curve. In studies of classification accuracy, there are often covariates that should be incorporated into the ROC analysis. We describe three different ways of using covariate information. For factors that affect marker observations among controls, we present a method for covariate adjustment. For factors that affect discrimination (i.e. the ROC curve), we describe methods for modelling the ROC curve as a function of covariates. Finally, for factors that contribute to discrimination, we propose combining the marker and covariate information, and ask how much discriminatory accuracy improves with the addition of the marker to the covariates (incremental value). These methods follow naturally when representing the ROC curve as a summary of the distribution of case marker observations, standardized with respect to the control distribution.

Assessment of Second-Opinion Strategies for Diagnoses of Cutaneous Melanocytic Lesions.

Importance: Histopathologic criteria have limited diagnostic reliability for a range of cutaneous melanocytic lesions. Objective: To evaluate the association of second-opinion strategies by general pathologists and dermatopathologists with the overall reliability of diagnosis of difficult melanocytic lesions. Design, Setting, and Participants: This diagnostic study used samples from the Melanoma Pathology Study, which comprises 240 melanocytic lesion samples selected from a dermatopathology laboratory in Bellevue, Washington, and represents the full spectrum of lesions from common nevi to invasive melanoma. Five sets of 48 samples were evaluated independently by 187 US pathologists from July 15, 2013, through May 23, 2016. Data analysis was performed from April 2016 through November 2017. Main Outcomes and Measures: Accuracy of diagnosis, defined as concordance with an expert consensus diagnosis of 3 experienced pathologists, was assessed after applying 10 different second-opinion strategies. Results: Among the 187 US pathologists examining the 24 lesion samples, 113 were general pathologists (65 men [57.5%]; mean age at survey, 53.7 years [range, 33.0-79.0 years]) and 74 were dermatopathologists (49 men [66.2%]; mean age at survey, 46.4 years [range, 33.0-77.0 years]). Among the 8976 initial case interpretations, physicians desired second opinions for 3899 (43.4%), most often for interpretation of severely dysplastic nevi. The overall misclassification rate was highest when interpretations did not include second opinions and initial reviewers were all general pathologists lacking subspecialty training (52.8%; 95% CI, 51.3%-54.3%). When considering different second opinion strategies, the misclassification of melanocytic lesions was lowest when the first, second, and third consulting reviewers were subspecialty-trained dermatopathologists and when all lesions were subject to second opinions (36.7%; 95% CI, 33.1%-40.7%). When the second opinion strategies were compared with single interpretations without second opinions, the reductions in misclassification rates for some of the strategies were statistically significant, but none of the strategies eliminated diagnostic misclassification. Melanocytic lesions in the middle of the diagnostic spectrum had the highest misclassification rates (eg, moderately or severely dysplastic nevus, Spitz nevus, melanoma in situ, and pathologic stage [p]T1a invasive melanoma). Variability of in situ and thin invasive melanoma was relatively intractable to all examined strategies. Conclusions and Relevance: The results of this study suggest that second opinions rendered by dermatopathologists improve reliability of melanocytic lesion diagnosis. However, discordance among pathologists remained high.

Integrating the predictiveness of a marker with its performance as a classifier.

There are two popular statistical approaches to biomarker evaluation. One models the risk of disease (or disease outcome) with, for example, logistic regression. A marker is considered useful if it has a strong effect on risk. The second evaluates classification performance by use of measures such as sensitivity, specificity, predictive values, and receiver operating characteristic curves. There is controversy about which approach is more appropriate. Moreover, the two approaches can give contradictory results on the same data. The authors present a new graphic, the predictiveness curve, which complements the risk modeling approach. It assesses the usefulness of a risk model when applied to the population. Although the predictiveness curve relates to classification performance measures, it also displays essential information about risk that is not displayed by the receiver operating characteristic curve. The authors propose that the predictiveness and classification performance of a marker, displayed together in an integrated plot, provide a comprehensive and cohesive assessment of a risk marker or model. The methods are demonstrated with data on prostate-specific antigen and risk factors from the Prostate Cancer Prevention Trial, 1993-2003.

Accuracy of Digital Pathologic Analysis vs Traditional Microscopy in the Interpretation of Melanocytic Lesions.

Importance: Use of digital whole-slide imaging (WSI) for dermatopathology in general has been noted to be similar to traditional microscopy (TM); however, concern has been noted that WSI is inferior for interpretation of melanocytic lesions. Since approximately 1 of every 4 skin biopsies is of a melanocytic lesion, the use of WSI requires verification before use in clinical practice. Objective: To compare pathologists' accuracy and reproducibility in diagnosing melanocytic lesions using Melanocytic Pathology Assessment Tool and Hierarchy for Diagnosis (MPATH-Dx) categories when analyzing by TM vs WSI. Design, Setting, and Participants: A total of 87 pathologists in community-based and academic settings from 10 US states were randomized with stratification based on clinical experience to interpret in TM format 180 skin biopsy cases of melanocytic lesions, including 90 invasive melanoma, divided into 5 sets of 36 cases (phase 1). The pathologists were then randomized via stratified permuted block randomization with block size 2 to interpret cases in either TM (n = 46) or WSI format (n = 41), with each pathologist interpreting the same 36 cases on 2 separate occasions (phase 2). Diagnoses were categorized as MPATH-Dx categories I through V, with I indicating the least severe and V the most severe. Main Outcomes and Measures: Accuracy with respect to a consensus reference diagnosis and the reproducibility of repeated interpretations of the same cases. Results: Of the 87 pathologists in the study, 46% (40) were women and the mean (SD) age was 50.7 (10.2) years. Except for class III melanocytic lesions, the diagnostic categories showed no significant differences in diagnostic accuracy between TM and WSI interpretation. Discordance was lower among class III lesions for the TM interpretation arm (51%; 95% CI, 46%-57%) than for the WSI arm (61%; 95% CI, 53%-69%) (P = .05). This difference is likely to have clinical significance, because 6% of TM vs 11% of WSI class III lesions were interpreted as invasive melanoma. Reproducibility was similar between the traditional and digital formats overall (66.4%; 95% CI, 63.3%-69.3%; and 62.7%; 95% CI, 59.5%-65.7%, respectively), and for all classes, although class III showed a nonsignificant lower intraobserver agreement for digital. Significantly more mitotic figures were detected with TM compared with WSI: mean (SD) TM, 6.72 (2.89); WSI, 5.84 (2.56); P = .002. Conclusions and Relevance: Interpretive accuracy for melanocytic lesions was similar for WSI and TM slides except for class III lesions. We found no clinically meaningful differences in reproducibility for any of the diagnostic classes.