Performance and estimation of the true error rate of classification rules built with additional information. An application to a cancer trial.

ABSTRACT

Classification rules that incorporate additional information usually present in discrimination problems are receiving certain attention during the last years as they perform better than the usual rules. Fernández, M. A., C. Rueda and B. Salvador (2006) "Incorporating additional information to normal linear discriminant rules," J. Am. Stat. Assoc., 101, 569-577, proved that these rules have lower total misclassification probability than the usual Fisher's rule. In this paper we consider two issues; on the one hand, we compare these rules with those based on shrinkage estimators of the mean proposed by Tong, T., L. Chen and H. Zhao (2012) "Improved mean estimation and its application to diagonal discriminant analysis," Bioinformatics, 28(4) 531-537. with regard to four criteria total misclassification probability, area under ROC curve, well-calibratedness and refinement; on the other hand, we consider the estimation of the true error rate, which is a very interesting parameter in applications. We prove results on the apparent error rate of the rules that expose the need of new estimators of their true error rate. We propose four such new estimators. Two of them are defined incorporating the additional information into the leave-one-out-bootstrap. The other two are the corresponding cross-validation after bootstrap versions. We compare these estimators with the usual ones in a simulation study and in a cancer trial application, showing the good behavior of the rules that incorporate additional information and of the new leave-one-out bootstrap estimators of their true error rate.