The impact of different sources of heterogeneity on loss of accuracy from genomic prediction models.

Cross-study validation (CSV) of prediction models is an alternative to traditional cross-validation (CV) in domains where multiple comparable datasets are available. Although many studies have noted potential sources of heterogeneity in genomic studies, to our knowledge none have systematically investigated their intertwined impacts on prediction accuracy across studies. We employ a hybrid parametric/non-parametric bootstrap method to realistically simulate publicly available compendia of microarray, RNA-seq, and whole metagenome shotgun microbiome studies of health outcomes. Three types of heterogeneity between studies are manipulated and studied: (i) imbalances in the prevalence of clinical and pathological covariates, (ii) differences in gene covariance that could be caused by batch, platform, or tumor purity effects, and (iii) differences in the "true" model that associates gene expression and clinical factors to outcome. We assess model accuracy, while altering these factors. Lower accuracy is seen in CSV than in CV. Surprisingly, heterogeneity in known clinical covariates and differences in gene covariance structure have very limited contributions in the loss of accuracy when validating in new studies. However, forcing identical generative models greatly reduces the within/across study difference. These results, observed consistently for multiple disease outcomes and omics platforms, suggest that the most easily identifiable sources of study heterogeneity are not necessarily the primary ones that undermine the ability to accurately replicate the accuracy of omics prediction models in new studies. Unidentified heterogeneity, such as could arise from unmeasured confounding, may be more important.

Ocular ischaemic complications in giant cell arteritis: CHADS2-score predicts risk of permanent visual impairment.

OBJECTIVES: To identify independent risk factors for permanent visual loss (PVL) in patients with giant cell arteritis (GCA), with a special focus on sonographic findings of the temporal, carotid and subclavian/axillary arteries, and on established scoring systems of ischaemia risk assessment. METHODS: Consecutive patients with a diagnosis of GCA between 2002 and 2013 were retrospectively identified from a prospectively maintained database. Data on clinical characteristics including ophthalmological findings, laboratory values, and sonographic findings of the temporal, carotid an axillary arteries were extracted. CHADS2- and CHA2DS2-VASc-score were calculated. Clinical, laboratory and sonographic characteristics of patients with and without PVL were compared. Multiple logistic regression models were calculated to identify variables independently associated with PVL. RESULTS: One-hundred-fifty-two patients were included in the analysis. PVL occurred in 30.2% of patients, with anterior ischaemic optic neuropathy as predominant underlying cause (91.3%). The frequency of PVL was strongly dependent on the age at diagnosis, with a significant increase after the age of 70 years. In multivariate analysis, axillary artery vasculitis with an odds ratio (OR) of 0.3 and constitutional symptoms with an OR of 0.1 were negatively associated with PVL. A CHADS2-score of 1 (OR 10.7) or ≥2 (OR 25) was associated with a significantly increased risk of PVL. CONCLUSIONS: The risk of PVL secondary to GCA increases with age but is lower in patients presenting with constitutional symptoms and/or exhibiting axillary artery involvement. The CHADS2-score may help to discriminate patients with low vs. high risk of PVL.

Improving cross-study prediction through addon batch effect adjustment or addon normalization.

Motivation: To date most medical tests derived by applying classification methods to high-dimensional molecular data are hardly used in clinical practice. This is partly because the prediction error resulting when applying them to external data is usually much higher than internal error as evaluated through within-study validation procedures. We suggest the use of addon normalization and addon batch effect removal techniques in this context to reduce systematic differences between external data and the original dataset with the aim to improve prediction performance. Results: We evaluate the impact of addon normalization and seven batch effect removal methods on cross-study prediction performance for several common classifiers using a large collection of microarray gene expression datasets, showing that some of these techniques reduce prediction error. Availability and Implementation: All investigated addon methods are implemented in our R package bapred. Contact: hornung@ibe.med.uni-muenchen.de. Supplementary information: Supplementary data are available at Bioinformatics online.

Cross-study validation for the assessment of prediction algorithms.

MOTIVATION: Numerous competing algorithms for prediction in high-dimensional settings have been developed in the statistical and machine-learning literature. Learning algorithms and the prediction models they generate are typically evaluated on the basis of cross-validation error estimates in a few exemplary datasets. However, in most applications, the ultimate goal of prediction modeling is to provide accurate predictions for independent samples obtained in different settings. Cross-validation within exemplary datasets may not adequately reflect performance in the broader application context. METHODS: We develop and implement a systematic approach to 'cross-study validation', to replace or supplement conventional cross-validation when evaluating high-dimensional prediction models in independent datasets. We illustrate it via simulations and in a collection of eight estrogen-receptor positive breast cancer microarray gene-expression datasets, where the objective is predicting distant metastasis-free survival (DMFS). We computed the C-index for all pairwise combinations of training and validation datasets. We evaluate several alternatives for summarizing the pairwise validation statistics, and compare these to conventional cross-validation. RESULTS: Our data-driven simulations and our application to survival prediction with eight breast cancer microarray datasets, suggest that standard cross-validation produces inflated discrimination accuracy for all algorithms considered, when compared to cross-study validation. Furthermore, the ranking of learning algorithms differs, suggesting that algorithms performing best in cross-validation may be suboptimal when evaluated through independent validation. AVAILABILITY: The survHD: Survival in High Dimensions package (http://www.bitbucket.org/lwaldron/survhd) will be made available through Bioconductor.

Impact of cranial and axillary/subclavian artery involvement by color duplex sonography on response to treatment in giant cell arteritis.

OBJECTIVE: Color duplex sonography (CDS) today is broadly used in the diagnostic workup of patients with suspected cranial or extracranial giant cell arteritis (GCA). This study aimed to determine the prognostic impact of the disease pattern assessed by CDS on the treatment response in GCA. METHODS: This was a retrospective, longitudinal follow-up study of 43 patients who were diagnosed with GCA at our institution between 2002 and 2010. All patients underwent CDS of the temporal and subclavian/axillary arteries at baseline and were observed for at least 6 months. Vasculitis was sonographically characterized by a circumferential, hypoechogenic wall thickening. According to the CDS findings, patients were categorized into patients with involvement of the subclavian/axillary arteries only (group A1, n = 17), patients with involvement of both the subclavian/axillary arteries and the temporal arteries (group A2, n = 9), and patients with isolated cranial GCA (group B, n = 17). Data on recurrences, corticosteroid doses, and steroid-sparing agents were extracted from the medical records. Treatment response over time was analyzed by Kaplan-Meier curves with log-rank testing. RESULTS: The mean follow-up time was 25.4 months and did not differ between groups (P = .4). Patients in group A1 were significantly younger than patients in groups A2 and B (P < .01). The interval between symptom onset and diagnosis was significantly longer in groups A1 and A2 compared with group B (P < .01). The number of recurrences per month was significantly higher in group A2 compared with group A1 and group B (A1, 0.07; A2, 0.13; B, 0.03; P < .01). Whereas there were no significant differences in the mean time until a daily prednisolone dose <10 mg was reached, patients in group A2 more frequently required steroid-sparing agents (A1, 24%; A2, 56%; B, 24%; P = .04). CONCLUSIONS: Extensive vascular involvement of both the temporal and subclavian/axillary arteries, as depicted by CDS, may be associated with a poor treatment response in GCA.

A measure of the impact of CV incompleteness on prediction error estimation with application to PCA and normalization.

BACKGROUND: In applications of supervised statistical learning in the biomedical field it is necessary to assess the prediction error of the respective prediction rules. Often, data preparation steps are performed on the dataset-in its entirety-before training/test set based prediction error estimation by cross-validation (CV)-an approach referred to as "incomplete CV". Whether incomplete CV can result in an optimistically biased error estimate depends on the data preparation step under consideration. Several empirical studies have investigated the extent of bias induced by performing preliminary supervised variable selection before CV. To our knowledge, however, the potential bias induced by other data preparation steps has not yet been examined in the literature. In this paper we investigate this bias for two common data preparation steps: normalization and principal component analysis for dimension reduction of the covariate space (PCA). Furthermore we obtain preliminary results for the following steps: optimization of tuning parameters, variable filtering by variance and imputation of missing values. METHODS: We devise the easily interpretable and general measure CVIIM ("CV Incompleteness Impact Measure") to quantify the extent of bias induced by incomplete CV with respect to a data preparation step of interest. This measure can be used to determine whether a specific data preparation step should, as a general rule, be performed in each CV iteration or whether an incomplete CV procedure would be acceptable in practice. We apply CVIIM to large collections of microarray datasets to answer this question for normalization and PCA. RESULTS: Performing normalization on the entire dataset before CV did not result in a noteworthy optimistic bias in any of the investigated cases. In contrast, when performing PCA before CV, medium to strong underestimates of the prediction error were observed in multiple settings. CONCLUSIONS: While the investigated forms of normalization can be safely performed before CV, PCA has to be performed anew in each CV split to protect against optimistic bias.

Shear bond strength of two adhesives to bovine dentin contaminated with various astringents.

PURPOSE: To investigate the in vitro shear bond strength of two adhesives to bovine dentin contaminated with various astringents. METHODS: 120 adult bovine incisors were collected and cut to obtain 240 specimens. The specimens were randomly divided into a self-etch adhesive group (N = 120) and a total-etch adhesive group (N = 120). Both of the groups were divided into the following six subgroups: the non-contamination group and the contamination groups 25% Al2(SO4)3 (Orbat sensitive), 25% AlCl3 (Racestyptine), 10% AlCl3 (Roeko Gingiva Liquid), 15.5% Fe2(SO4)3 (Astringedent) and AlCl3 Paste (Astringent Retraction Paste, N = 20 in each subgroup). Each astringent was applied for 1 minute to the dentin surface before rinsing with water spray for 20 seconds. The respective adhesive was then applied according to the manufacturer's instructions. Two composite cylinders were shaped with a mold, cured on the dentin surface of each specimen and sheared off after 1 day and 1 week storage. The shear bond strengths (MPa) were recorded and analyzed with ANOVA. Results: In the self-etching adhesive group, all astringents showed negative effects on dentin bonding (P < 0.05). Astringent contamination did not have a negative effect on dentin bonding in the total-etch adhesive group (P > 0.05).

Correcting the optimal resampling-based error rate by estimating the error rate of wrapper algorithms.

High-dimensional binary classification tasks, for example, the classification of microarray samples into normal and cancer tissues, usually involve a tuning parameter. By reporting the performance of the best tuning parameter value only, over-optimistic prediction errors are obtained. For correcting this tuning bias, we develop a new method which is based on a decomposition of the unconditional error rate involving the tuning procedure, that is, we estimate the error rate of wrapper algorithms as introduced in the context of internal cross-validation (ICV) by Varma and Simon (2006, BMC Bioinformatics 7, 91). Our subsampling-based estimator can be written as a weighted mean of the errors obtained using the different tuning parameter values, and thus can be interpreted as a smooth version of ICV, which is the standard approach for avoiding tuning bias. In contrast to ICV, our method guarantees intuitive bounds for the corrected error. Additionally, we suggest to use bias correction methods also to address the conceptually similar method selection bias that results from the optimal choice of the classification method itself when evaluating several methods successively. We demonstrate the performance of our method on microarray and simulated data and compare it to ICV. This study suggests that our approach yields competitive estimates at a much lower computational price.

Impact of the postthrombotic syndrome on quality of life after primary upper extremity deep venous thrombosis.

BACKGROUND: To determine the impact of the postthrombotic syndrome (PTS) on quality of life after primary upper extremity deep venous thrombosis (UEDVT). PATIENTS AND METHODS: Twenty-five patients with a history of primary UEDVT, treated with anticoagulation alone, and twenty healthy controls were retrospectively identified and prospectively assessed for health-related quality of life (SF-36 and VEINES-QOL-questionnaire) and upper extremity functional impairment (DASH-score). Presence of PTS was classified according to the modified Villalta-score. Comparisons between patients and controls and between patients with and without PTS were performed using Fisher`s exact test (categorical variables) and Mann-Whitney-U-test (continuous variables). RESULTS: According to the modified Villalta-score, 32 % of the patients suffered from mild to moderate PTS. None of the patients developed severe PTS. Compared to healthy control subjects, patients with a history of primary UEDVT reported on considerably worse health-related quality of life and significantly stronger upper extremity functional impairment. Within the cohort of patients with UEDVT, subjects with PTS had a significantly reduced quality of life and a more severe functional limitation. CONCLUSIONS: Quality of life and functional performance are impaired in patients with a history of conservatively treated primary UEDVT. Impairment is most pronounced in patients with mild to moderate PTS occurring in every third patient.

A Diagnostic Algorithm Based on a Simple Clinical Prediction Rule for the Diagnosis of Cranial Giant Cell Arteritis.

BACKGROUND: Risk stratification based on pre-test probability may improve the diagnostic accuracy of temporal artery high-resolution compression sonography (hrTCS) in the diagnostic workup of cranial giant cell arteritis (cGCA). METHODS: A logistic regression model with candidate items was derived from a cohort of patients with suspected cGCA (n = 87). The diagnostic accuracy of the model was tested in the derivation cohort and in an independent validation cohort (n = 114) by receiver operator characteristics (ROC) analysis. The clinical items were composed of a clinical prediction rule, integrated into a stepwise diagnostic algorithm together with C-reactive protein (CRP) values and hrTCS values. RESULTS: The model consisted of four clinical variables (age > 70, headache, jaw claudication, and anterior ischemic optic neuropathy). The diagnostic accuracy of the model for discrimination of patients with and without a final clinical diagnosis of cGCA was excellent in both cohorts (area under the curve (AUC) 0.96 and AUC 0.92, respectively). The diagnostic algorithm improved the positive predictive value of hrCTS substantially. Within the algorithm, 32.8% of patients (derivation cohort) and 49.1% (validation cohort) would not have been tested by hrTCS. None of these patients had a final diagnosis of cGCA. CONCLUSION: A diagnostic algorithm based on a clinical prediction rule improves the diagnostic accuracy of hrTCS.

Embolic Protection in Complex Femoropopliteal Interventions: Safety, Efficacy and Predictors of Filter Macroembolization.

OBJECTIVES: To evaluate the safety and efficacy of a filter embolic protection device (FEPD) in endovascular interventions of the femoropopliteal arteries. METHODS: Patients who underwent endovascular interventions of the femoropopliteal arteries between 2008 and 2016 and in whom the SpiderFXTM FEPD was applied were included in this retrospective study. Clinical and angiographic characteristics, filter macroembolization (FME), device-related complications, distal embolization, as well as the early clinical and hemodynamic outcome, were assessed. Potential risk factors for FME were evaluated by multivariate analysis. RESULTS: A total of 244 cases were identified (203 patients, claudication 60.4%, critical limb ischaemia 39.6%, mean lesion length 13.2 ± 12.9 cm, complete occlusions in 72.7%). Balloon angioplasty ± stenting (BAP), directional atherectomy ± balloon angioplasty ± stenting (DA) and rotational thrombectomy ± balloon angioplasty ± stenting (RT) were performed in 141, 61 and 42 cases, respectively. FEPD placement and retrieval were successful in all but one case each. Permanent filter-related vessel damage was not observed. The rate of FME was 37.3% (BAP 36.2%, DA 32.8%, RT 47.7%). Risk factors for FME in the BAP- and DA-group were total occlusion, lesion length > 19 cm, visible thrombus and diabetes mellitus. The distal embolization rate despite filter protection was 4.1 % (BAP 4.9%, DA 1.6%, RT 4.8%) and was higher in cases with FME compared with those without FME (8.7% vs. 1.5%, p = 0.02). CONCLUSION: The Spider FXTM device is safe and effective in capturing embolic debris during femoropopliteal interventions. A residual risk of peripheral embolization remains. LEVEL OF EVIDENCE: III, Cohort study.

Comparative meta-analysis of prognostic gene signatures for late-stage ovarian cancer.

BACKGROUND: Ovarian cancer is the fifth most common cause of cancer deaths in women in the United States. Numerous gene signatures of patient prognosis have been proposed, but diverse data and methods make these difficult to compare or use in a clinically meaningful way. We sought to identify successful published prognostic gene signatures through systematic validation using public data. METHODS: A systematic review identified 14 prognostic models for late-stage ovarian cancer. For each, we evaluated its 1) reimplementation as described by the original study, 2) performance for prognosis of overall survival in independent data, and 3) performance compared with random gene signatures. We compared and ranked models by validation in 10 published datasets comprising 1251 primarily high-grade, late-stage serous ovarian cancer patients. All tests of statistical significance were two-sided. RESULTS: Twelve published models had 95% confidence intervals of the C-index that did not include the null value of 0.5; eight outperformed 97.5% of signatures including the same number of randomly selected genes and trained on the same data. The four top-ranked models achieved overall validation C-indices of 0.56 to 0.60 and shared anticorrelation with expression of immune response pathways. Most models demonstrated lower accuracy in new datasets than in validation sets presented in their publication. CONCLUSIONS: This analysis provides definitive support for a handful of prognostic models but also confirms that these require improvement to be of clinical value. This work addresses outstanding controversies in the ovarian cancer literature and provides a reproducible framework for meta-analytic evaluation of gene signatures.