Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review.

BACKGROUND: Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models are fit for purpose and remain relevant in the long-term. We aimed to present an overview of methodological guidance for the evaluation (i.e., validation and impact assessment) and updating of clinical prediction models. METHODS: We systematically searched nine databases from January 2000 to January 2022 for articles in English with methodological recommendations for the post-derivation stages of interest. Qualitative analysis was used to summarize the 70 selected guidance papers. RESULTS: Key aspects for validation are the assessment of statistical performance using measures for discrimination (e.g., C-statistic) and calibration (e.g., calibration-in-the-large and calibration slope). For assessing impact or usefulness in clinical decision-making, recent papers advise using decision-analytic measures (e.g., the Net Benefit) over simplistic classification measures that ignore clinical consequences (e.g., accuracy, overall Net Reclassification Index). Commonly recommended methods for model updating are recalibration (i.e., adjustment of intercept or baseline hazard and/or slope), revision (i.e., re-estimation of individual predictor effects), and extension (i.e., addition of new markers). Additional methodological guidance is needed for newer types of updating (e.g., meta-model and dynamic updating) and machine learning-based models. CONCLUSION: Substantial guidance was found for model evaluation and more conventional updating of regression-based models. An important development in model evaluation is the introduction of a decision-analytic framework for assessing clinical usefulness. Consensus is emerging on methods for model updating.

Prediction models for patients with esophageal or gastric cancer: A systematic review and meta-analysis.

BACKGROUND: Clinical prediction models are increasingly used to predict outcomes such as survival in cancer patients. The aim of this study was threefold. First, to perform a systematic review to identify available clinical prediction models for patients with esophageal and/or gastric cancer. Second, to evaluate sources of bias in the included studies. Third, to investigate the predictive performance of the prediction models using meta-analysis. METHODS: MEDLINE, EMBASE, PsycINFO, CINAHL, and The Cochrane Library were searched for publications from the year 2000 onwards. Studies describing models predicting survival, adverse events and/or health-related quality of life (HRQoL) for esophageal or gastric cancer patients were included. Potential sources of bias were assessed and a meta-analysis, pooled per prediction model, was performed on the discriminative abilities (c-indices). RESULTS: A total of 61 studies were included (45 development and 16 validation studies), describing 47 prediction models. Most models predicted survival after a curative resection. Nearly 75% of the studies exhibited bias in at least 3 areas and model calibration was rarely reported. The meta-analysis showed that the averaged c-index of the models is fair (0.75) and ranges from 0.65 to 0.85. CONCLUSION: Most available prediction models only focus on survival after a curative resection, which is only relevant to a limited patient population. Few models predicted adverse events after resection, and none focused on patient's HRQoL, despite its relevance. Generally, the quality of reporting is poor and external model validation is limited. We conclude that there is a need for prediction models that better meet patients' information needs, and provide information on both the benefits and harms of the various treatment options in terms of survival, adverse events and HRQoL.

Decision support systems for incurable non-small cell lung cancer: a systematic review.

BACKGROUND: Individually tailored cancer treatment is essential to ensure optimal treatment and resource use. Treatments for incurable metastatic non-small cell lung cancer (NSCLC) are evolving rapidly, and decision support systems (DSS) for this patient population have been developed to balance benefits and harms for decision-making. The aim of this systematic review was to inventory DSS for stage IIIB/IV NSCLC patients. METHODS: A systematic literature search was performed in Pubmed, Embase and the Cochrane Library. DSS were described extensively, including their predictors, model performances (i.e., discriminative ability and calibration), levels of validation and user friendliness. RESULTS: The systematic search yielded 3531 articles. In total, 67 articles were included after additional reference tracking. The 39 identified DSS aim to predict overall survival and/or progression-free survival, but give no information about toxicity or cost-effectiveness. Various predictors were incorporated, such as performance status, serum and inflammatory markers, and patient and tumor characteristics. Some DSS were developed for the entire incurable NSCLC population, whereas others were specifically for patients with brain or spinal metastases. Few DSS had been validated externally using recent clinical data, and the discrimination and calibration were often poor. CONCLUSIONS: Many DSS have been developed for incurable NSCLC patients, but DSS are still lacking that are up-to-date with a good model performance, while covering the entire treatment spectrum. Future DSS should incorporate genetic and biological markers based on state-of-the-art evidence, and compare multiple treatment options to estimate survival, toxicity and cost-effectiveness.

Oncologist, patient, and companion questions during pretreatment consultations about adjuvant cancer treatment: a shared decision-making perspective.

OBJECTIVES: To assess the occurrence of questions that foster shared decision making, in particular cancer patients' understanding of treatment decisions and oncologists' understanding of patients' priorities, during consultations in which preference-sensitive decisions are discussed. Specifically, (a) regarding patient understanding, do oncologists ask about patients' preexisting knowledge, information preferences, and understanding and do patients and companions ask about the disease and treatment, and (b) regarding patient priorities, do oncologists ask about patients' treatment- and decision-related preferences and do patients and companions ask about the decision? METHODS: Audiotaped pretreatment consultations of 100 cancer patients with 32 oncologists about (neo)adjuvant treatment were coded and analyzed to document question type, topic, and initiative. RESULTS: The oncologists ascertained prior knowledge in 50 patients, asked 24 patients about preferred (probability) information, and invited questions from 56 patients. The oncologists asked 32 patients about treatment preferences and/or for consent. Respectively, one-third and one-fifth of patients and companions asked about treatment benefits compared with three-quarters of them who asked about treatment harms and/or procedures. CONCLUSIONS: It would be helpful to patients if oncologists more often assessed patients' existing knowledge to tailor their information provision. Also, patients could receive treatment recommendations that better fit their personal situation if oncologists collected information on patients' views about treatments. Moreover, by educating patients to ask about treatment alternatives, benefits, and harms, patients may gain a better understanding of the choice they have.

Timing of risk reducing mastectomy in breast cancer patients carrying a BRCA1/2 mutation: retrospective data from the Dutch HEBON study.

It is expected that rapid genetic counseling and testing (RGCT) will lead to increasing numbers of breast cancer (BC) patients knowing their BRCA1/2 carrier status before primary surgery. Considering the potential impact of knowing one's status on uptake and timing of risk-reducing contralateral mastectomy (RRCM), we aimed to evaluate trends over time in RRCM, and differences between carriers identified either before (predictively) or after (diagnostically) diagnosis. We collected data from female BRCA1/2 mutation carriers diagnosed with BC between 1995 and 2009 from four Dutch university hospitals. We compared the timing of genetic testing and RRCM in relation to diagnosis in 1995-2000 versus 2001-2009 for all patients, and predictively and diagnostically tested patients separately. Of 287 patients, 219 (76%) had a diagnostic BRCA1/2 test. In this cohort, the median time from diagnosis to DNA testing decreased from 28 months for those diagnosed between 1995 and 2000 to 14 months for those diagnosed between 2001 and 2009 (p < 0.001). Similarly, over time women in this cohort underwent RRCM sooner after diagnosis (median of 77 vs. 27 months, p = 0.05). Predictively tested women who subsequently developed BC underwent an immediate RRCM significantly more often than women who had a diagnostic test (21/61, 34%, vs. 13/170, 7.6 %, p < 0.001). Knowledge of carrying a BRCA1/2 mutation when diagnosed with BC influenced decisions concerning primary surgery. Additionally, in more recent years, women who had not undergone predictive testing were more likely to undergo diagnostic DNA testing and RRCM sooner after diagnosis. This suggests the need for RGCT to guide treatment decisions.