GRADE concept paper 8: judging the certainty of discrimination performance estimates of prognostic models in a body of validation studies.

BACKGROUND: Prognostic models incorporate multiple prognostic factors to estimate the likelihood of future events for individual patients based on their prognostic factor values. Evaluating these models crucially involves conducting studies to assess their predictive performance, like discrimination. Systematic reviews and meta-analyses of these validation studies play an essential role in selecting models for clinical practice. METHODS: In this paper, we outline 3 thresholds to determine the target for certainty rating in the discrimination of prognostic models, as observed across a body of validation studies. RESULTS AND CONCLUSION: We propose 3 thresholds when rating the certainty of evidence about a prognostic model's discrimination. The first threshold amounts to rating certainty in the model's ability to classify better than random chance. The other 2 approaches involve setting thresholds informed by other mechanisms for classification: clinician intuition or an alternative prognostic model developed for the same disease area and outcome. The choice of threshold will vary based on the context. Instead of relying on arbitrary discrimination cut-offs, our approach positions the observed discrimination within an informed spectrum, potentially aiding decisions about a prognostic model's practical utility.

GRADE guidance 36: updates to GRADE's approach to addressing inconsistency.

OBJECTIVES: To update previous Grading of Recommendations Assessment, Development and Evaluation (GRADE) guidance by addressing inconsistencies and interpreting subgroup analyses. STUDY DESIGN AND SETTING: Using an iterative process, we consulted with members of the GRADE working group through multiple rounds of written feedback and discussions at GRADE working group meetings. RESULTS: The guidance complements previous guidance with clarification in two areas: (1) assessing inconsistency and (2) assessing the credibility of possible effect modifiers that might explain inconsistency. Specifically, the guidance clarifies that inconsistency refers to variability in results, not in study characteristics; that inconsistency assessment for binary outcomes requires consideration of both relative and absolute effects; how to decide between narrower and broader questions in systematic reviews and guidelines; that, with the same evidence, ratings of inconsistency may differ depending on the target of certainty rating; and how GRADE inconsistency ratings relate to a statistical measure of inconsistency I2 depending on the context in which one views results. The second part of the guidance illustrates, based on a worked example, the use of the instrument to assess the credibility of effect modification analyses. The guidance explains the stepwise process of moving from a subgroup analysis to assessing the credibility of effect modification and, if found credible, to subgroup-specific effect estimates and GRADE certainty ratings. CONCLUSION: This updated guidance addresses specific conceptual and practical issues that systematic review authors frequently face when considering the degree of inconsistency in estimates of treatment effects across studies.

GRADE concept paper 2: Concepts for judging certainty on the calibration of prognostic models in a body of validation studies.

BACKGROUND: Prognostic models combine several prognostic factors to provide an estimate of the likelihood (or risk) of future events in individual patients, conditional on their prognostic factor values. A fundamental part of evaluating prognostic models is undertaking studies to determine whether their predictive performance, such as calibration and discrimination, is reproduced across settings. Systematic reviews and meta-analyses of studies evaluating prognostic models' performance are a necessary step for selection of models for clinical practice and for testing the underlying assumption that their use will improve outcomes, including patient's reassurance and optimal future planning. METHODS: In this paper, we highlight key concepts in evaluating the certainty of evidence regarding the calibration of prognostic models. RESULTS AND CONCLUSION: Four concepts are key to evaluating the certainty of evidence on prognostic models' performance regarding calibration. The first concept is that the inference regarding calibration may take one of two forms: deciding whether one is rating certainty that a model's performance is satisfactory or, instead, unsatisfactory, in either case defining the threshold for satisfactory (or unsatisfactory) model performance. Second, inconsistency is the critical GRADE domain to deciding whether we are rating certainty in the model performance being satisfactory or unsatisfactory. Third, depending on whether one is rating certainty in satisfactory or unsatisfactory performance, different patterns of inconsistency of results across studies will inform ratings of certainty of evidence. Fourth, exploring the distribution of point estimates of observed to expected ratio across individual studies, and its determinants, will bear on the need for and direction of future research.

GRADE Guidelines 28: Use of GRADE for the assessment of evidence about prognostic factors: rating certainty in identification of groups of patients with different absolute risks.

OBJECTIVE: The objective of this study was to provide guidance on the use of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to determine certainty in estimates of association between prognostic factors and future outcomes. STUDY DESIGN AND SETTING: We developed our guidance through an iterative process that involved review of published systematic reviews and meta-analyses of prognostic factors, consultation with members, feedback, presentation, and discussion at the GRADE Working Group meetings. RESULTS: For questions of prognosis, a body of observational evidence (potentially including patients enrolled in randomized controlled trials) begins as high certainty in the evidence. The five domains of GRADE for rating down certainty in the evidence, that is, risk of bias, imprecision, inconsistency, indirectness, and publication bias, as well as the domains for rating up, also apply to estimates of associations between prognostic factors and outcomes. One should determine if their ratings do not consider (noncontextualized) or consider (contextualized) the clinical context as this will may result in variable judgments on certainty of the evidence. CONCLUSIONS: The same principles GRADE proposed for bodies of evidence addressing treatment and overall prognosis work well in assessing individual prognostic factors, both in noncontextualized and contextualized settings.