High certainty evidence is stable and trustworthy, whereas evidence of moderate or lower certainty may be equally prone to being unstable.

OBJECTIVE: To assess to what extent the overall quality of evidence indicates changes to observed intervention effect estimates when new data become available. STUDY DESIGN AND SETTING: We conducted a meta- epidemiological study. We obtained evidence from meta-analyses of randomized trials of Cochrane reviews addressing the same healthcare question that was updated with inclusion of additional data between January 2016 and May 2021. METHODS: We extracted the reported effect estimates with 95% confidence intervals from meta-analyses and corresponding GRADE (Grading of Recommendations Assessment, Development, and Evaluation) assessments of any intervention comparison for the primary outcome in the first and the last updated review version. We considered the reported overall quality (certainty) of evidence (CoE) and specific evidence limitations (no, serious or very serious for risk of bias, imprecision, inconsistency, and/or indirectness). We assessed the change in pooled effect estimates between the original and updated evidence using the ratio of odds ratio (ROR), absolute ROR (aROR), ratio of standard errors (RoSE), direction of effects, and level of statistical significance. RESULTS: High CoE without limitations characterized 19.3% (n=29) out of 150 included original Cochrane reviews. The update with additional data did not systematically change the effect estimates (mean ROR 1.00; 95%CI 0.99-1.02), which deviated 1.06-fold from the older estimates (median aROR; IQR: 1.01-1.15), gained precision (median RoSE 0.87; IQR 0.76-1.00), and maintained the same direction with the same level of statistical significance in 93% (27 of 29) of cases. Lower CoE with limitations characterized 121 original reviews and graded as moderate CoE in 30.0% (45 of 150), low CoE in 32.0% (48 of 150), and very low CoE in 18.7% (28 of 150) reviews. Their update had larger absolute deviations (median aROR 1.12 to 1.33) and larger gains in precision (median RoSE 0.78 to 0.86) without clear and consistent differences between these categories of CoE. Changes in effect direction or statistical significance were also more common in the lower quality evidence, again with a similar extent across categories (without change in 75.6%, 64.6%, and 75.0% for moderate, low, very low CoE). As limitations increased, effect estimates deviated more (aROR 1.05 with zero, 1.11 with one, 1.25 with two, 1.24 with three limitations) and changes in direction or significance became more frequent (93.2% stable with no limitations, 74.5% with one, 68.2% with two, and 61.5% with three limitations). CONCLUSIONS: High-quality evidence without methodological deficiencies is trustworthy and stable, providing reliable intervention effect estimates when updated with new data. Evidence of moderate and lower quality may be equally prone to being unstable and cannot indicate if available effect estimates are true, exaggerated, or underestimated.

Converting IMPROVE bleeding and VTE risk assessment models into a fast-and-frugal decision tree for VTE prophylaxis.

Current hospital VTE (venous thromboembolism) prophylaxis for medical patients has been characterized by both underuse and, increasingly, overuse. The American Society of Hematology (ASH) has endorsed the use of risk assessment models (RAM) as an approach to individualize VTE prophylaxis as a way of balancing overuse (excessive risk of bleeding) and underuse (risk of avoidable VTE). ASH has endorsed IMPROVE (International Medical Prevention Registry on Venous Thromboembolism) risk assessment models - the only RAM to assess short-term bleeding and VTE risk in acutely ill medical inpatients. ASH, however, notes that no RAMs have been thoroughly analyzed for their effect on patient outcomes. We aimed to validate the IMPROVE models and adapt them into a simple, fast-and-frugal (FFT) decision tree to evaluate the impact of VTE prevention on health outcomes and costs. We employed three methods: the "best evidence" from ASH guidelines, a "learning health system paradigm" combining guideline and real-world data from the Medical University of South Carolina (MUSC), and a "real-world data" approach based solely on MUSC data retrospectively extracted from electronic records. We found the most effective VTE prevention strategy utilizes the FFT decision tree, guided by an IMPROVE VTE score of ≥ 2 or ≥ 4 and a bleeding score of < 7. This method could prevent 45% of unnecessary treatments, saving about $5 million annually for patients like the MUSC cohort. We recommend integrating the IMPROVE models into hospital electronic medical records as a point-of-care tool, thereby enhancing VTE prevention in hospitalized medical patients.

Decision theoretical foundation of clinical practice guidelines: an extension of the ASH thrombophilia guidelines.

Decision analysis can play an essential role in informing practice guidelines. The American Society of Hematology (ASH) thrombophilia guidelines have made a significant step forward in demonstrating how decision modeling integrated within GRADE (Grading of Recommendations Assessment, Developing, and Evaluation) methodology can advance the field of guideline development. Although the ASH model was transparent and understandable, it does, however, suffer from the certain limitations that may have generated potentially wrong recommendations. That is, the panel considered two models separately- after 3-6 months of index venous thromboembolism (VTE), the panel compared Thrombophilia Testing (A) vs. discontinuing anticoagulants (B) and Test (A) vs C (recommending indefinite anticoagulation to all patients) instead of considering all relevant options simultaneously (A vs. B vs. C). Our study aimed to avoid what we refer to as the omitted choice bias by integrating two ASH models into a single unifying threshold decision model. We analyzed 6 ASH panel's recommendations related to testing for thrombophilia in settings of "provoked" vs. "unprovoked" venous thromboembolism (VTE) and low vs. high-bleeding risk (total 12 recommendations). Our model disagreed with the ASH guidelines panels' recommendations in 4 of the 12 recommendations we considered. Considering all three options simultaneously, our model provided results that would have produced sounder recommendations for patient care. By revisiting the ASH guidelines methodology, we have not only improved recommendations for thrombophilia but also provided a method that can be easily applied to other clinical problems and promises to improve the current guidelines' methodology.

Distinguishing ASH Clinical Practice Guidelines from Other Forms of ASH Clinical Advice.

The American Society of Hematology (ASH) develops a variety of resources that provide guidance to clinicians on the diagnosis and management of blood diseases. These resources include clinical practice guidelines (CPGs) and other forms of clinical advice. While both ASH CPGs and other forms of clinical advice provide recommendations, they differ with respect to the methods underpinning their development, the principal type of recommendations they offer, their transparency and concordance with published evidence, and the time and resources required for their development. It is crucial that end users be aware of the differences between CPGs and other forms of clinical advice and that producers and publishers of these resources use clear and unambiguous terminology to facilitate their distinction. The objective of this article is to highlight similarities and differences between ASH CPGs and other forms of ASH clinical advice and to discuss the implications of these differences for end users.

Methods proposed for monitoring the implementation of evidence-based research: a cross-sectional study.

OBJECTIVES: Evidence-based research (EBR) is the systematic and transparent use of prior research to inform a new study so that it answers questions that matter in a valid, efficient, and accessible manner. This study surveyed experts about existing (e.g., citation analysis) and new methods for monitoring EBR and collected ideas about implementing these methods. STUDY DESIGN AND SETTING: We conducted a cross-sectional study via an online survey between November 2022 and March 2023. Participants were experts from the fields of evidence synthesis and research methodology in health research. Open-ended questions were coded by recurring themes; descriptive statistics were used for quantitative questions. RESULTS: Twenty-eight expert participants suggested that citation analysis should be supplemented with content evaluation (not just what is cited but also in which context), content expert involvement, and assessment of the quality of cited systematic reviews. They also suggested that citation analysis could be facilitated with automation tools. They emphasized that EBR monitoring should be conducted by ethics committees and funding bodies before the research starts. Challenges identified for EBR implementation monitoring were resource constraints and clarity on responsibility for EBR monitoring. CONCLUSION: Ideas proposed in this study for monitoring the implementation of EBR can be used to refine methods and define responsibility but should be further explored in terms of feasibility and acceptability. Different methods may be needed to determine if the use of EBR is improving over time.

Improving methods of clinical practice guidelines: From guidelines to pathways to fast-and-frugal trees and decision analysis to develop individualised patient care.

BACKGROUND: Current methods for developing clinical practice guidelines have several limitations: they are characterised by the "black box" operation-a process with defined inputs and outputs but an incomplete understanding of its internal workings; they have "the integration problem"-a lack of framework for explicitly integrating factors such as patient preferences and trade-offs between benefits and harms; they generate one recommendation at a time that typically are not connected in a coherent analytical framework; and they apply to "average" patients, while clinicians and their patients seek advice tailored to individual circumstances. METHODS: We propose augmenting the current guideline development method by converting evidence-based pathways into fast-and-frugal decision trees (FFTs) and integrating them with generalised decision curve analysis to formulate clear, individualised management recommendations. RESULTS: We illustrate the process by developing recommendations for the management of heparin-induced thrombocytopenia (HIT). We converted evidence-based pathways for HIT, developed by the American Society of Hematology, into an FFT. Here, we consider only thrombotic complications and major bleeding. We leveraged the predictive potential of FFTs to compare the effects of argatroban, bivalirudin, fondaparinux, and direct oral anticoagulants (DOACs) using generalised decision curve analysis. We found that DOACs were superior to other treatments if the FFT-predicted probability of HIT exceeded 3%. CONCLUSIONS: The proposed analytical framework connects guidelines, pathways, FFTs, and decision analysis, offering risk-tailored personalised recommendations and addressing current guideline development critiques.

Decision curve analysis based on summary data.

BACKGROUND: To realize the potential of precision medicine, predictive models should be integrated within the framework of decision analysis, such as the decision curve analysis (DCA). To date, its application has required individual patient data (IPD) that are often unavailable. Performing DCA using aggregate data without requiring IPD may advance the goals of precision medicine. METHODS: We present a statistical framework demonstrating that DCA can be conducted by using only the mean and standard deviation (SD) from the raw probabilities of the predictive model. We tested our theoretical framework by performing extensive simulations and comparing the aggregate-based DCA with IPD DCA. The latter was conducted using IPD from four predictive models that employed logistic regression, Cox or competing risk time-to-event modeling including (a) statins for primary prevention of cardiovascular disease (n = 4859), (b) hospice referral for terminally ill patients (n = 9104), (c) use of thromboprophylaxis for preventing venous thromboembolism in patients with cancer (n = 425) and (d) prevention of sinusoidal obstruction syndrome after hematopoietic cell transplantation (SCT) (n = 80). RESULTS: Simulations assuming perfect calibration showed that regardless of which probability distributions informed the predictive models, the differences in DCA were negligible. Similarly, for the adequately powered models, the results of DCA based on the summary data were similar to IPD-derived DCA. The inherent instability of the predictive models, based on the smaller sample sizes, resulted in a somewhat larger discrepancy between aggregate and IPD-based DCA. CONCLUSIONS: DCA informed by adequately powered and well-calibrated models using only summary statistical estimates (mean and SD) approximates well models using IPD. Use of aggregate data will facilitate broader integration of predictive with decision modeling toward the goals of individualized decision-making.

Guideline panel social dynamics influence the development of clinical practice recommendations: a mixed-methods systematic review.

OBJECTIVES: To synthesize empirical studies that investigate the cognitive and social processes involved in the deliberation process of guideline development meetings and determine the distribution of deliberated topics. STUDY DESIGN AND SETTING: We conducted a mixed-method systematic review using a convergent segregated approach. We searched for empirical studies that investigate the intragroup dynamics of guideline development meetings indexed in bibliographic databases. RESULTS: Of the 5,899 citations screened, 12 studies from six countries proved eligible. Chairs, cochairs, and methodologists contributed to at least one-third of the discussion time in guideline development meetings; patient partners contributed the least. In interdisciplinary groups, male gender and occupation as a physician were positively associated with the amount of contribution. Compared to groups that used the Grading of Recommendations Assessment, Development and Evaluation approach, for groups that did not, when faced with insufficient or low-quality evidence, relied more on their clinical experience. The presence of a cognitive "yes" bias was apparent in meetings: panelists tended to acquiesce with positive statements that required less cognitive effort than negative statements. CONCLUSION: The social dynamics of the discussions were linked to each panelist's activity role, professional background, and gender, all of which influenced the level of contributions they made in guideline development meetings.

Making Decisions When No Further Diagnostic Testing is Available.

In this chapter, we illustrate how evidence about treatments' benefits and harms can be integrated to enable rational decision-making even under considerable clinical uncertainty.

Evidence and Decision-Making.

Today, every country struggles to provide adequate health care to its citizens. Globally, an average of $8.3 trillion or 10% of gross domestic product (GDP) is annually spent on health services. In 2019, the USA spent nearly 18% ($3.2 trillion) of its GDP on health care, projected to reach $6.2 trillion by 2028.

Making Decisions When no Further Diagnostic Testing is Available (Expected Regret Theory Threshold Model).

In Chap. 2 , we illustrated the application of the expected utility theory (EUT) to rational decision-making when no further diagnostic testing is available. In this chapter, we apply regret theory to the decision problems discussed in Chap. 2 .

Decision-Making When Diagnostic Testing is Available.

When a decision-maker has the option of diagnostic testing, they face a typical dilemma: (1) do not administer treatment and do not test, (2) test and decide to administer treatment based on the test result, and (3) administer treatment without testing. In this chapter, we will discuss the theory behind threshold modeling when diagnostic testing is available; we will illustrate the approach by presenting a case vignette.

Formulating Management Strategies Using Fast-and-Frugal Trees (A Decision Tool to Transform Clinical Practice Guidelines and Clinical Pathways into Decision Support at the Point of Care).

Clinical management is rarely based on the collection of one data item. Instead, it is typically characterized by the continuous collection and evaluation of clinical data (symptoms, signs, laboratory, imaging tests, etc.) to establish a platform for further management decisions.

Using Decision Curve Analysis to Evaluate Testing and/or Predictive Modeling.

In this chapter, we extend the threshold model to evaluate the value of diagnostic tests or predictive models over a range of all possible thresholds by using decision curve analysis (DCA). DCA has been developed within the expected utility theory (EUT) and expected regret theory (ERT) framework.

Hybrid and Dual-Processing Threshold Decision Models.

In the previous chapters, we presented various derivations of the threshold model based on the same disease outcomes. We assumed that a decision-maker would calculate the threshold based on either mortality or morbidity outcomes. Basinga and van den Ende derived the threshold by combining both mortality and morbidity outcomes.

Medical Decision-Making and Artificial Intelligence.

In this chapter, we discuss the potential role that artificial intelligence (AI) may have in medical decision-making, the pros and cons, and the limitations and biases that might be introduced when using these novel techniques. As computing becomes more powerful and models continue to grow increasingly more complex, the potential of AI to improve decision-making is increasingly promising. Within many medical fields, however, at the time of this writing (September 2023), the promise of AI is yet to translate into everyday reality. Here, we summarize the role of AI in medical decision-making (diagnosis, prognosis, and treatment).

Which Threshold Model?

As outlined in the Preface (and Chap. 1 and other chapters), this book espoused two fundamental views. The first view consists of the proposal that the threshold model represents a method to address the Sorites paradox, which is a consequence of a relationship between scientific evidence (that exists on a continuum of credibility) and decision-making (that is categorical, yes/no exercises).