Evaluation of polygenic risk models using multiple performance measures: a critical assessment of discordant results.

PURPOSE: The area under the receiver operating characteristic curve (AUC) is commonly used for evaluating the improvement of polygenic risk models and increasingly assessed together with the net reclassification improvement (NRI) and integrated discrimination improvement (IDI). We evaluated how researchers described and interpreted AUC, NRI, and IDI when simultaneously assessed. METHODS: We reviewed how researchers described definitions of AUC, NRI, and IDI and how they computed each metric. Next, we reviewed how the increment in AUC, NRI, and IDI were interpreted, and how the overall conclusion about the improvement of the risk model was reached. RESULTS: AUC, NRI, and IDI were correctly defined in 63, 70, and 0% of the articles. All statistically significant values and almost half of the nonsignificant were interpreted as indicative of improvement, irrespective of the values of the metrics. Also, small, nonsignificant changes in the AUC were interpreted as indication of improvement when NRI and IDI were statistically significant. CONCLUSION: Researchers have insufficient knowledge about how to interpret the various metrics for the assessment of the predictive performance of polygenic risk models and rely on the statistical significance for their interpretation. A better understanding is needed to achieve more meaningful interpretation of polygenic prediction studies.

Reflection on modern methods: Revisiting the area under the ROC Curve.

The area under the receiver operating characteristic (ROC) curve (AUC) is commonly used for assessing the discriminative ability of prediction models even though the measure is criticized for being clinically irrelevant and lacking an intuitive interpretation. Every tutorial explains how the coordinates of the ROC curve are obtained from the risk distributions of diseased and non-diseased individuals, but it has not become common sense that therewith the ROC plot is just another way of presenting these risk distributions. We show how the ROC curve is an alternative way to present risk distributions of diseased and non-diseased individuals and how the shape of the ROC curve informs about the overlap of the risk distributions. For example, ROC curves are rounded when the prediction model included variables with similar effect on disease risk and have an angle when, for example, one binary risk factor has a stronger effect; and ROC curves are stepped rather than smooth when the sample size or incidence is low, when the prediction model is based on a relatively small set of categorical predictors. This alternative perspective on the ROC plot invalidates most purported limitations of the AUC and attributes others to the underlying risk distributions. AUC is a measure of the discriminative ability of prediction models. The assessment of prediction models should be supplemented with other metrics to assess their clinical utility.

DPD Testing Before Treatment With Fluoropyrimidines in the Amsterdam UMCs: An Evaluation of Current Pharmacogenetic Practice.

INTRODUCTION: The fluoropyrimidines (FP) (5-Fluorouracil, capecitabine, and tegafur) are commonly used anti-cancer drugs, but lead to moderate to severe toxicity in about 10-40% of patients. DPD testing [either the enzyme activity of dihydropyrimidine dehydrogenase (DPD) or the DPYD genotype] identifies patients at higher risk for toxicity who may be treated more safely with a lower drug dose. The Netherland's National guideline for colon carcinoma was updated in 2017 to recommend DPYD genotyping before treatment with FP. Pretreatment DPYD genotyping identifies approximately 50% of the patients that will develop severe FP toxicity. The aim of the study was to assess the uptake of DPD testing in the Amsterdam University Medical Centers over time and to evaluate stakeholder experiences to indicate barriers and facilitators of implementation in routine clinical care. MATERIALS AND METHODS: We used a mixed-method approach involving electronic patient records of 753 unique patients and pharmacy information systems analyses and fifteen semi-structured interviews with oncologists, pharmacists, and patients. The constellation perspective was used to identify barriers and facilitators at the level of practice, culture and structure. The proportion of FP users who were DPD tested pretreatment showed an increase from 1% (1/86) in Q2-2017 up to 87% (73/84) in Q4-2018. Unlike a landmark paper published in 2015, the National guideline for colorectal carcinoma followed by meetings to achieve local consensus led to this steep increase in the proportion of patients tested. RESULTS: Facilitating factors for stakeholders to implement testing included the existence of clear protocols, (anecdotal) evidence of the utility, being aware that peers are adhering to standard practice and clear and simple procedures for ordering and reporting. Main barriers included the lack of clear divisions of responsibilities, the lack of consensus on a test approach, long turn-around times and non-user-friendly IT-infrastructures. More professional education on the utility and limitations of pharmacogenetic testing was desired by most stakeholders. CONCLUSION: While the evidence for DPD testing was sufficient, only after the update of a National guideline and local consensus meetings the proportion of FP users that were DPD tested pretreatment rose to 87%. The implementation of personalized medicine requires stakeholders involved to attune practice, culture and structure.

Small improvement in the area under the receiver operating characteristic curve indicated small changes in predicted risks.

OBJECTIVE: Adding risk factors to a prediction model often increases the area under the receiver operating characteristic curve (AUC) only slightly, particularly when the AUC of the model was already high. We investigated whether a risk factor that minimally improves the AUC may nevertheless improve the predictive ability of the model, assessed by integrated discrimination improvement (IDI). STUDY DESIGN AND SETTING: We simulated data sets with risk factors and event status for 100,000 hypothetical individuals and created prediction models with AUCs between 0.50 and 0.95. We added a single risk factor for which the effect was modeled as a certain odds ratio (OR 2, 4, 8) or AUC increment (ΔAUC 0.01, 0.02, 0.03). RESULTS: Across all AUC values of the baseline model, for a risk factor with the same OR, both ΔAUC and IDI were lower when the AUC of the baseline model was higher. When the increment in AUC was small (ΔAUC 0.01), the IDI was also small, except when the AUC of the baseline model was >0.90. CONCLUSION: When the addition of a risk factor shows minimal improvement in AUC, predicted risks generally show minimal changes too. Updating risk models with strong risk factors may be informative for a subgroup of individuals, but not at the population level. The AUC may not be as insensitive as is frequently argued.