A spline-based tool to assess and visualize the calibration of multiclass risk predictions.
J Biomed Inform
; 54: 283-93, 2015 Apr.
Article
in En
| MEDLINE
| ID: mdl-25579635
ABSTRACT
When validating risk models (or probabilistic classifiers), calibration is often overlooked. Calibration refers to the reliability of the predicted risks, i.e. whether the predicted risks correspond to observed probabilities. In medical applications this is important because treatment decisions often rely on the estimated risk of disease. The aim of this paper is to present generic tools to assess the calibration of multiclass risk models. We describe a calibration framework based on a vector spline multinomial logistic regression model. This framework can be used to generate calibration plots and calculate the estimated calibration index (ECI) to quantify lack of calibration. We illustrate these tools in relation to risk models used to characterize ovarian tumors. The outcome of the study is the surgical stage of the tumor when relevant and the final histological outcome, which is divided into five classes benign, borderline malignant, stage I, stage II-IV, and secondary metastatic cancer. The 5909 patients included in the study are randomly split into equally large training and test sets. We developed and tested models using the following algorithms logistic regression, support vector machines, k nearest neighbors, random forest, naive Bayes and nearest shrunken centroids. Multiclass calibration plots are interesting as an approach to visualizing the reliability of predicted risks. The ECI is a convenient tool for comparing models, but is less informative and interpretable than calibration plots. In our case study, logistic regression and random forest showed the highest degree of calibration, and the naive Bayes the lowest.
Key words
Full text:
1
Collection:
01-internacional
Database:
MEDLINE
Main subject:
Models, Statistical
/
Risk Assessment
/
Decision Support Systems, Clinical
Type of study:
Etiology_studies
/
Prognostic_studies
/
Risk_factors_studies
Limits:
Adult
/
Aged
/
Female
/
Humans
/
Middle aged
Language:
En
Journal:
J Biomed Inform
Journal subject:
INFORMATICA MEDICA
Year:
2015
Document type:
Article
Affiliation country: