Assessing calibration of multinomial risk prediction models.

Calibration, that is, whether observed outcomes agree with predicted risks, is important when evaluating risk prediction models. For dichotomous outcomes, several tools exist to assess different aspects of model calibration, such as calibration-in-the-large, logistic recalibration, and (non-)parametric calibration plots. We aim to extend these tools to prediction models for polytomous outcomes. We focus on models developed using multinomial logistic regression (MLR): outcome Y with k categories is predicted using k - 1 equations comparing each category i (i = 2, … ,k) with reference category 1 using a set of predictors, resulting in k - 1 linear predictors. We propose a multinomial logistic recalibration framework that involves an MLR fit where Y is predicted using the k - 1 linear predictors from the prediction model. A non-parametric alternative may use vector splines for the effects of the linear predictors. The parametric and non-parametric frameworks can be used to generate multinomial calibration plots. Further, the parametric framework can be used for the estimation and statistical testing of calibration intercepts and slopes. Two illustrative case studies are presented, one on the diagnosis of malignancy of ovarian tumors and one on residual mass diagnosis in testicular cancer patients treated with cisplatin-based chemotherapy. The risk prediction models were developed on data from 2037 and 544 patients and externally validated on 1107 and 550 patients, respectively. We conclude that calibration tools can be extended to polytomous outcomes. The polytomous calibration plots are particularly informative through the visual summary of the calibration performance.

A comparison between an ultrasound based prediction model (LR2) and the risk of ovarian malignancy algorithm (ROMA) to assess the risk of malignancy in women with an adnexal mass.

OBJECTIVE: The identification of novel biomarkers led to the development of the ROMA algorithm incorporating both HE4 and CA125 to predict malignancy in women with a pelvic mass. An ultrasound based prediction model (LR2) developed by the International Ovarian Tumor Analysis (IOTA) study offers better diagnostic performance than CA125 alone. In this study we compared the diagnostic accuracy between LR2 and ROMA. METHODS: This study included women with a pelvic mass scheduled for surgery and enrolled in a previous prospective diagnostic accuracy study. Experienced ultrasound examiners, general gynecologists and trainees supervised by one of the experts performed the preoperative transvaginal ultrasound examinations. Serum biomarkers were taken prior to surgery. Accuracy of LR2 and ROMA was estimated at completion of this study and did not form part of the decision making process. Final outcome was histology of removed tissues and surgical stage if relevant. RESULTS: In total 360 women were evaluated. 216 women had benign disease and 144 a malignancy. Overall test performance of LR2 (AUC 0.952) with 94% sensitivity and 82% specificity was significantly better than ROMA (AUC 0.893) with 84% sensitivity and 80% specificity. Difference in AUC was 0.059 (95% CI: 0.026-0.091; P-value 0.0004). Similar results were obtained when stratified for menopausal status. CONCLUSION: LR2 shows a better diagnostic performance than ROMA for the characterization of a pelvic mass in both pre- and postmenopausal women. These findings suggest that HE4 and CA125 may not play an important role in the diagnosis of ovarian cancer if good quality ultrasonography is available.

High-Throughput Gene Expression Profiles to Define Drug Similarity and Predict Compound Activity.

By adding biological information, beyond the chemical properties and desired effect of a compound, uncharted compound areas and connections can be explored. In this study, we add transcriptional information for 31K compounds of Janssen's primary screening deck, using the HT L1000 platform and assess (a) the transcriptional connection score for generating compound similarities, (b) machine learning algorithms for generating target activity predictions, and (c) the scaffold hopping potential of the resulting hits. We demonstrate that the transcriptional connection score is best computed from the significant genes only and should be interpreted within its confidence interval for which we provide the stats. These guidelines help to reduce noise, increase reproducibility, and enable the separation of specific and promiscuous compounds. The added value of machine learning is demonstrated for the NR3C1 and HSP90 targets. Support Vector Machine models yielded balanced accuracy values ≥80% when the expression values from DDIT4 & SERPINE1 and TMEM97 & SPR were used to predict the NR3C1 and HSP90 activity, respectively. Combining both models resulted in 22 new and confirmed HSP90-independent NR3C1 inhibitors, providing two scaffolds (i.e., pyrimidine and pyrazolo-pyrimidine), which could potentially be of interest in the treatment of depression (i.e., inhibiting the glucocorticoid receptor (i.e., NR3C1), while leaving its chaperone, HSP90, unaffected). As such, the initial hit rate increased by a factor 300, as less, but more specific chemistry could be screened, based on the upfront computed activity predictions.

Genetically Engineered iPSC-Derived FTDP-17 MAPT Neurons Display Mutation-Specific Neurodegenerative and Neurodevelopmental Phenotypes.

Tauopathies such as frontotemporal dementia (FTD) remain incurable to date, partially due to the lack of translational in vitro disease models. The MAPT gene, encoding the microtubule-associated protein tau, has been shown to play an important role in FTD pathogenesis. Therefore, we used zinc finger nucleases to introduce two MAPT mutations into healthy donor induced pluripotent stem cells (iPSCs). The IVS10+16 mutation increases the expression of 4R tau, while the P301S mutation is pro-aggregant. Whole-transcriptome analysis of MAPT IVS10+16 neurons reveals neuronal subtype differences, reduced neural progenitor proliferation potential, and aberrant WNT/SHH signaling. Notably, these neurodevelopmental phenotypes could be recapitulated in neurons from patients carrying the MAPT IVS10+16 mutation. Moreover, the additional pro-aggregant P301S mutation revealed additional phenotypes, such as an increased calcium burst frequency, reduced lysosomal acidity, tau oligomerization, and neurodegeneration. This series of iPSCs could serve as a platform to unravel a potential link between pathogenic 4R tau and FTD.

Validation and updating of risk models based on multinomial logistic regression.

BACKGROUND: Risk models often perform poorly at external validation in terms of discrimination or calibration. Updating methods are needed to improve performance of multinomial logistic regression models for risk prediction. METHODS: We consider simple and more refined updating approaches to extend previously proposed methods for dichotomous outcomes. These include model recalibration (adjustment of intercept and/or slope), revision (re-estimation of individual model coefficients), and extension (revision with additional markers). We suggest a closed testing procedure to assist in deciding on the updating complexity. These methods are demonstrated on a case study of women with pregnancies of unknown location (PUL). A previously developed risk model predicts the probability that a PUL is a failed, intra-uterine, or ectopic pregnancy. We validated and updated this model on more recent patients from the development setting (temporal updating; n = 1422) and on patients from a different hospital (geographical updating; n = 873). Internal validation of updated models was performed through bootstrap resampling. RESULTS: Contrary to dichotomous models, we noted that recalibration can also affect discrimination for multinomial risk models. If the number of outcome categories is higher than the number of variables, logistic recalibration is obsolete because straightforward model refitting does not require the estimation of more parameters. Although recalibration strongly improved performance in the case study, the closed testing procedure selected model revision. Further, revision of functional form of continuous predictors had a positive effect on discrimination, whereas penalized estimation of changes in model coefficients was beneficial for calibration. CONCLUSIONS: Methods for updating of multinomial risk models are now available to improve predictions in new settings. A closed testing procedure is helpful to decide whether revision is preferred over recalibration. Because multicategory outcomes increase the number of parameters to be estimated, we recommend full model revision only when the sample size for each outcome category is large.

Presurgical diagnosis of adnexal tumours using mathematical models and scoring systems: a systematic review and meta-analysis.

BACKGROUND Characterizing ovarian pathology is fundamental to optimizing management in both pre- and post-menopausal women. Inappropriate referral to oncology services can lead to unnecessary surgery or overly radical interventions compromising fertility in young women, whilst the consequences of failing to recognize cancer significantly impact on prognosis. By reflecting on recent developments of new diagnostic tests for preoperative identification of malignant disease in women with adnexal masses, we aimed to update a previous systematic review and meta-analysis. METHODS An extended search was performed in MEDLINE (PubMed) and EMBASE (OvidSp) from March 2008 to October 2013. Eligible studies provided information on diagnostic test performance of models, designed to predict ovarian cancer in a preoperative setting, that contained at least two variables. Study selection and extraction of study characteristics, types of bias, and test performance was performed independently by two reviewers. Quality was assessed using a modified version of the QUADAS assessment tool. A bivariate hierarchical random effects model was used to produce summary estimates of sensitivity and specificity with 95% confidence intervals or plot summary ROC curves for all models considered. RESULTS Our extended search identified a total of 1542 new primary articles. In total, 195 studies were eligible for qualitative data synthesis, and 96 validation studies reporting on 19 different prediction models met the predefined criteria for quantitative data synthesis. These models were tested on 26 438 adnexal masses, including 7199 (27%) malignant and 19 239 (73%) benign masses. The Risk of Malignancy Index (RMI) was the most frequently validated model. The logistic regression model LR2 with a risk cut-off of 10% and Simple Rules (SR), both developed by the International Ovarian Tumor Analysis (IOTA) study, performed better than all other included models with a pooled sensitivity and specificity, respectively, of 0.92 [95% CI 0.88-0.95] and 0.83 [95% CI 0.77-0.88] for LR2 and 0.93 [95% CI 0.89-0.95] and 0.81 [95% CI 0.76-0.85] for SR. A meta-analysis of centre-specific results stratified for menopausal status of two multicentre cohorts comparing LR2, SR and RMI-1 (using a cut-off of 200) showed a pooled sensitivity and specificity in premenopausal women for LR2 of 0.85 [95% CI 0.75-0.91] and 0.91 [95% CI 0.83-0.96] compared with 0.93 [95% CI 0.84-0.97] and 0.83 [95% CI 0.73-0.90] for SR and 0.44 [95% CI 0.28-0.62] and 0.95 [95% CI 0.90-0.97] for RMI-1. In post-menopausal women, sensitivity and specificity of LR2, SR and RMI-1 were 0.94 [95% CI 0.89-0.97] and 0.70 [95% CI 0.62-0.77], 0.93 [95% CI 0.88-0.96] and 0.76 [95% CI 0.69-0.82], and 0.79 [95% CI 0.72-0.85] and 0.90 [95% CI 0.84-0.94], respectively. CONCLUSIONS An evidence-based approach to the preoperative characterization of any adnexal mass should incorporate the use of IOTA Simple Rules or the LR2 model, particularly for women of reproductive age.

Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study.

OBJECTIVES: To develop a risk prediction model to preoperatively discriminate between benign, borderline, stage I invasive, stage II-IV invasive, and secondary metastatic ovarian tumours. DESIGN: Observational diagnostic study using prospectively collected clinical and ultrasound data. SETTING: 24 ultrasound centres in 10 countries. PARTICIPANTS: Women with an ovarian (including para-ovarian and tubal) mass and who underwent a standardised ultrasound examination before surgery. The model was developed on 3506 patients recruited between 1999 and 2007, temporally validated on 2403 patients recruited between 2009 and 2012, and then updated on all 5909 patients. MAIN OUTCOME MEASURES: Histological classification and surgical staging of the mass. RESULTS: The Assessment of Different NEoplasias in the adneXa (ADNEX) model contains three clinical and six ultrasound predictors: age, serum CA-125 level, type of centre (oncology centres v other hospitals), maximum diameter of lesion, proportion of solid tissue, more than 10 cyst locules, number of papillary projections, acoustic shadows, and ascites. The area under the receiver operating characteristic curve (AUC) for the classic discrimination between benign and malignant tumours was 0.94 (0.93 to 0.95) on temporal validation. The AUC was 0.85 for benign versus borderline, 0.92 for benign versus stage I cancer, 0.99 for benign versus stage II-IV cancer, and 0.95 for benign versus secondary metastatic. AUCs between malignant subtypes varied between 0.71 and 0.95, with an AUC of 0.75 for borderline versus stage I cancer and 0.82 for stage II-IV versus secondary metastatic. Calibration curves showed that the estimated risks were accurate. CONCLUSIONS: The ADNEX model discriminates well between benign and malignant tumours and offers fair to excellent discrimination between four types of ovarian malignancy. The use of ADNEX has the potential to improve triage and management decisions and so reduce morbidity and mortality associated with adnexal pathology.

Simple dichotomous updating methods improved the validity of polytomous prediction models.

OBJECTIVES: Prediction models may perform poorly in a new setting. We aimed to determine which model updating methods should be applied for models predicting polytomous outcomes, which often suffer from one or more categories with low prevalence. STUDY DESIGN AND SETTING: We used case studies on testicular and ovarian tumors. The original regression models were based on 502 and 2,037 patients and validated on 273 and 1,107 patients, respectively. The polytomous models combined dichotomous models for category A vs. B + C and B vs. C (sequential dichotomous modeling). Simple recalibration, revision, and redevelopment methods were considered. To assess discrimination (using dichotomous and polytomous c-statistics) and calibration (by comparing observed and expected prevalences) of these methods, the validation data were divided into updating and test parts. Five hundred such divisions were randomly generated, and the average test set results reported. RESULTS: None of the updating methods could improve discrimination of the original models, but recalibration, revision, and redevelopment strongly improved calibration. Redevelopment was unstable with respect to overfitting and performance. CONCLUSION: Simple dichotomous updating methods behaved well when applied to polytomous models. Our results suggest that recalibration is preferred, but larger validation sets may make revision or redevelopment a sensible alternative.