Understanding random resampling techniques for class imbalance correction and their consequences on calibration and discrimination of clinical risk prediction models.

OBJECTIVE: Class imbalance is sometimes considered a problem when developing clinical prediction models and assessing their performance. To address it, correction strategies involving manipulations of the training dataset, such as random undersampling or oversampling, are frequently used. The aim of this article is to illustrate the consequences of these class imbalance correction strategies on clinical prediction models' internal validity in terms of calibration and discrimination performances. METHODS: We used both heuristic intuition and formal mathematical reasoning to characterize the relations between conditional probabilities of interest and probabilities targeted when using random undersampling or oversampling. We propose a plug-in estimator that represents a natural correction for predictions obtained from models that have been trained on artificially balanced datasets ("naïve" models). We conducted a Monte Carlo simulation with two different data generation processes and present a real-world example using data from the International Stroke Trial database to empirically demonstrate the consequences of applying random resampling techniques for class imbalance correction on calibration and discrimination (in terms of Area Under the ROC, AUC) for logistic regression and tree-based prediction models. RESULTS: Across our simulations and in the real-world example, calibration of the naïve models was very poor. The models using the plug-in estimator generally outperformed the models relying on class imbalance correction in terms of calibration while achieving the same discrimination performance. CONCLUSION: Random resampling techniques for class imbalance correction do not generally improve discrimination performance (i.e., AUC), and their use is hard to justify when aiming at providing calibrated predictions. Improper use of such class imbalance correction techniques can lead to suboptimal data usage and less valid risk prediction models.

Should Cognitive Screening Tests Be Corrected for Age and Education? Insights From a Causal Perspective.

Cognitive screening tests such as the Mini-Mental State Examination are widely used in clinical routine to predict cognitive impairment. The raw test scores are often corrected for age and education, although documented poorer discrimination performance of corrected scores has challenged this practice. Nonetheless, test correction persists, perhaps due to the seemingly counterintuitive nature of the underlying problem. We used a causal framework to inform the long-standing debate from a more intuitive angle. We illustrate and quantify the consequences of applying the age-education correction of cognitive tests on discrimination performance. In an effort to bridge theory and practical implementation, we computed differences in discrimination performance under plausible causal scenarios using Open Access Series of Imaging Studies (OASIS)-1 data. We show that when age and education are causal risk factors for cognitive impairment and independently also affect the test score, correcting test scores for age and education removes meaningful information, thereby diminishing discrimination performance.

MRI-targeted biopsy cores from prostate index lesions: assessment and prediction of the number needed.

BACKGROUND: Magnetic resonance imaging (MRI) is used to detect the prostate index lesion before targeted biopsy. However, the number of biopsy cores that should be obtained from the index lesion is unclear. The aim of this study is to analyze how many MRI-targeted biopsy cores are needed to establish the most relevant histopathologic diagnosis of the index lesion and to build a prediction model. METHODS: We retrospectively included 451 patients who underwent 10-core systematic prostate biopsy and MRI-targeted biopsy with sampling of at least three cores from the index lesion. A total of 1587 biopsy cores were analyzed. The core sampling sequence was recorded, and the first biopsy core detecting the most relevant histopathologic diagnosis was identified. In a subgroup of 261 patients in whom exactly three MRI-targeted biopsy cores were obtained from the index lesion, we generated a prediction model. A nonparametric Bayes classifier was trained using the PI-RADS score, prostate-specific antigen (PSA) density, lesion size, zone, and location as covariates. RESULTS: The most relevant histopathologic diagnosis of the index lesion was detected by the first biopsy core in 331 cases (73%), by the second in 66 cases (15%), and by the third in 39 cases (9%), by the fourth in 13 cases (3%), and by the fifth in two cases (<1%). The Bayes classifier correctly predicted which biopsy core yielded the most relevant histopathologic diagnosis in 79% of the subjects. PI-RADS score, PSA density, lesion size, zone, and location did not independently influence the prediction model. CONCLUSION: The most relevant histopathologic diagnosis of the index lesion was made on the basis of three MRI-targeted biopsy cores in 97% of patients. Our classifier can help in predicting the first MRI-targeted biopsy core revealing the most relevant histopathologic diagnosis; however, at least three MRI-targeted biopsy cores should be obtained regardless of the preinterventionally assessed covariates.

Therapeutic implications of improved molecular diagnostics for rare CNS embryonal tumor entities: results of an international, retrospective study.

BACKGROUND: Only few data are available on treatment-associated behavior of distinct rare CNS embryonal tumor entities previously treated as "CNS-primitive neuroectodermal tumors" (CNS-PNET). Respective data on specific entities, including CNS neuroblastoma, FOXR2 activated (CNS NB-FOXR2), and embryonal tumors with multilayered rosettes (ETMR) are needed for development of differentiated treatment strategies. METHODS: Within this retrospective, international study, tumor samples of clinically well-annotated patients with the original diagnosis of CNS-PNET were analyzed using DNA methylation arrays (n = 307). Additional cases (n = 66) with DNA methylation pattern of CNS NB-FOXR2 were included irrespective of initial histological diagnosis. Pooled clinical data (n = 292) were descriptively analyzed. RESULTS: DNA methylation profiling of "CNS-PNET" classified 58 (19%) cases as ETMR, 57 (19%) as high-grade glioma (HGG), 36 (12%) as CNS NB-FOXR2, and 89(29%) cases were classified into 18 other entities. Sixty-seven (22%) cases did not show DNA methylation patterns similar to established CNS tumor reference classes. Best treatment results were achieved for CNS NB-FOXR2 patients (5-year PFS: 63% ± 7%, OS: 85% ± 5%, n = 63), with 35/42 progression-free survivors after upfront craniospinal irradiation (CSI) and chemotherapy. The worst outcome was seen for ETMR and HGG patients with 5-year PFS of 18% ± 6% and 22% ± 7%, and 5-year OS of 24% ± 6% and 25% ± 7%, respectively. CONCLUSION: The historically reported poor outcome of CNS-PNET patients becomes highly variable when tumors are molecularly classified based on DNA methylation profiling. Patients with CNS NB-FOXR2 responded well to current treatments and a standard-risk CSI-based regimen may be prospectively evaluated. The poor outcome of ETMR across applied treatment strategies substantiates the necessity for evaluation of novel treatments.