Accounting for treatment use when validating a prognostic model: a simulation study.
BMC Med Res Methodol
; 17(1): 103, 2017 Jul 14.
Article
em En
| MEDLINE
| ID: mdl-28709404
ABSTRACT
BACKGROUND:
Prognostic models often show poor performance when applied to independent validation data sets. We illustrate how treatment use in a validation set can affect measures of model performance and present the uses and limitations of available analytical methods to account for this using simulated data.METHODS:
We outline how the use of risk-lowering treatments in a validation set can lead to an apparent overestimation of risk by a prognostic model that was developed in a treatment-naïve cohort to make predictions of risk without treatment. Potential methods to correct for the effects of treatment use when testing or validating a prognostic model are discussed from a theoretical perspective.. Subsequently, we assess, in simulated data sets, the impact of excluding treated individuals and the use of inverse probability weighting (IPW) on the estimated model discrimination (c-index) and calibration (observedexpected ratio and calibration plots) in scenarios with different patterns and effects of treatment use.RESULTS:
Ignoring the use of effective treatments in a validation data set leads to poorer model discrimination and calibration than would be observed in the untreated target population for the model. Excluding treated individuals provided correct estimates of model performance only when treatment was randomly allocated, although this reduced the precision of the estimates. IPW followed by exclusion of the treated individuals provided correct estimates of model performance in data sets where treatment use was either random or moderately associated with an individual's risk when the assumptions of IPW were met, but yielded incorrect estimates in the presence of non-positivity or an unobserved confounder.CONCLUSIONS:
When validating a prognostic model developed to make predictions of risk without treatment, treatment use in the validation set can bias estimates of the performance of the model in future targeted individuals, and should not be ignored. When treatment use is random, treated individuals can be excluded from the analysis. When treatment use is non-random, IPW followed by the exclusion of treated individuals is recommended, however, this method is sensitive to violations of its assumptions.
Texto completo:
1
Coleções:
01-internacional
Base de dados:
MEDLINE
Assunto principal:
Avaliação de Resultados em Cuidados de Saúde
/
Modelos Teóricos
Tipo de estudo:
Prognostic_studies
Limite:
Humans
Idioma:
En
Revista:
BMC Med Res Methodol
Assunto da revista:
MEDICINA
Ano de publicação:
2017
Tipo de documento:
Article
País de afiliação:
Holanda