High certainty evidence is stable and trustworthy, whereas evidence of moderate or lower certainty may be equally prone to being unstable.

OBJECTIVES: To assess to what extent the overall quality of evidence indicates changes to observe intervention effect estimates when new data become available. METHODS: We conducted a meta-epidemiological study. We obtained evidence from meta-analyses of randomized trials of Cochrane reviews addressing the same health-care question that was updated with inclusion of additional data between January 2016 and May 2021. We extracted the reported effect estimates with 95% confidence intervals (CIs) from meta-analyses and corresponding GRADE (Grading of Recommendations Assessment, Development, and Evaluation) assessments of any intervention comparison for the primary outcome in the first and the last updated review version. We considered the reported overall quality (certainty) of evidence (CoE) and specific evidence limitations (no, serious or very serious for risk of bias, imprecision, inconsistency, and/or indirectness). We assessed the change in pooled effect estimates between the original and updated evidence using the ratio of odds ratio (ROR), absolute ratio of odds ratio (aROR), ratio of standard errors (RoSE), direction of effects, and level of statistical significance. RESULTS: High CoE without limitations characterized 19.3% (n = 29) out of 150 included original Cochrane reviews. The update with additional data did not systematically change the effect estimates (mean ROR 1.00; 95% CI 0.99-1.02), which deviated 1.06-fold from the older estimates (median aROR; interquartile range [IQR]: 1.01-1.15), gained precision (median RoSE 0.87; IQR 0.76-1.00), and maintained the same direction with the same level of statistical significance in 93% (27 of 29) of cases. Lower CoE with limitations characterized 121 original reviews and graded as moderate CoE in 30.0% (45 of 150), low CoE in 32.0% (48 of 150), and very low CoE in 18.7% (28 of 150) reviews. Their update had larger absolute deviations (median aROR 1.12 to 1.33) and larger gains in precision (median RoSE 0.78-0.86) without clear and consistent differences between these categories of CoE. Changes in effect direction or statistical significance were also more common in the lower quality evidence, again with a similar extent across categories (without change in 75.6%, 64.6%, and 75.0% for moderate, low, very low CoE). As limitations increased, effect estimates deviated more (aROR 1.05 with zero, 1.11 with one, 1.25 with two, 1.24 with three limitations) and changes in direction or significance became more frequent (93.2% stable with no limitations, 74.5% with one, 68.2% with two, and 61.5% with three limitations). CONCLUSION: High-quality evidence without methodological deficiencies is trustworthy and stable, providing reliable intervention effect estimates when updated with new data. Evidence of moderate and lower quality may be equally prone to being unstable and cannot indicate if available effect estimates are true, exaggerated, or underestimated.

Decision theoretical foundations of clinical practice guidelines: an extension of the ASH thrombophilia guidelines.

ABSTRACT: Decision analysis can play an essential role in informing practice guidelines. The American Society of Hematology (ASH) thrombophilia guidelines have made a significant step forward in demonstrating how decision modeling integrated within Grading of Recommendations Assessment, Developing, and Evaluation (GRADE) methodology can advance the field of guideline development. Although the ASH model was transparent and understandable, it does, however, suffer from certain limitations that may have generated potentially wrong recommendations. That is, the panel considered 2 models separately: after 3 to 6 months of index venous thromboembolism (VTE), the panel compared thrombophilia testing (A) vs discontinuing anticoagulants (B) and testing (A) vs recommending indefinite anticoagulation to all patients (C), instead of considering all relevant options simultaneously (A vs B vs C). Our study aimed to avoid what we refer to as the omitted choice bias by integrating 2 ASH models into a single unifying threshold decision model. We analyzed 6 ASH panel's recommendations related to the testing for thrombophilia in settings of "provoked" vs "unprovoked" VTE and low vs high bleeding risk (total 12 recommendations). Our model disagreed with the ASH guideline panels' recommendations in 4 of the 12 recommendations we considered. Considering all 3 options simultaneously, our model provided results that would have produced sounder recommendations for patient care. By revisiting the ASH guidelines methodology, we have not only improved the recommendations for thrombophilia but also provided a method that can be easily applied to other clinical problems and promises to improve the current guidelines' methodology.

Converting IMPROVE bleeding and VTE risk assessment models into a fast-and-frugal decision tree for optimal hospital VTE prophylaxis.

ABSTRACT: Current hospital venous thromboembolism (VTE) prophylaxis for medical patients is characterized by both underuse and overuse. The American Society of Hematology (ASH) has endorsed the use of risk assessment models (RAMs) as an approach to individualize VTE prophylaxis by balancing overuse (excessive risk of bleeding) and underuse (risk of avoidable VTE). ASH has endorsed IMPROVE (International Medical Prevention Registry on Venous Thromboembolism) risk assessment models, the only RAMs to assess short-term bleeding and VTE risk in acutely ill medical inpatients. ASH, however, notes that no RAMs have been thoroughly analyzed for their effect on patient outcomes. We aimed to validate the IMPROVE models and adapt them into a simple, fast-and-frugal (FFT) decision tree to evaluate the impact of VTE prevention on health outcomes and costs. We used 3 methods: the "best evidence" from ASH guidelines, a "learning health system paradigm" combining guideline and real-world data from the Medical University of South Carolina (MUSC), and a "real-world data" approach based solely on MUSC data retrospectively extracted from electronic records. We found that the most effective VTE prevention strategy used the FFT decision tree based on an IMPROVE VTE score of ≥2 or ≥4 and a bleeding score of <7. This method could prevent 45% of unnecessary treatments, saving ∼$5 million annually for patients such as the MUSC cohort. We recommend integrating IMPROVE models into hospital electronic medical records as a point-of-care tool, thereby enhancing VTE prevention in hospitalized medical patients.

Decision curve analysis based on summary data.

BACKGROUND: To realize the potential of precision medicine, predictive models should be integrated within the framework of decision analysis, such as the decision curve analysis (DCA). To date, its application has required individual patient data (IPD) that are often unavailable. Performing DCA using aggregate data without requiring IPD may advance the goals of precision medicine. METHODS: We present a statistical framework demonstrating that DCA can be conducted by using only the mean and standard deviation (SD) from the raw probabilities of the predictive model. We tested our theoretical framework by performing extensive simulations and comparing the aggregate-based DCA with IPD DCA. The latter was conducted using IPD from four predictive models that employed logistic regression, Cox or competing risk time-to-event modeling including (a) statins for primary prevention of cardiovascular disease (n = 4859), (b) hospice referral for terminally ill patients (n = 9104), (c) use of thromboprophylaxis for preventing venous thromboembolism in patients with cancer (n = 425) and (d) prevention of sinusoidal obstruction syndrome after hematopoietic cell transplantation (SCT) (n = 80). RESULTS: Simulations assuming perfect calibration showed that regardless of which probability distributions informed the predictive models, the differences in DCA were negligible. Similarly, for the adequately powered models, the results of DCA based on the summary data were similar to IPD-derived DCA. The inherent instability of the predictive models, based on the smaller sample sizes, resulted in a somewhat larger discrepancy between aggregate and IPD-based DCA. CONCLUSIONS: DCA informed by adequately powered and well-calibrated models using only summary statistical estimates (mean and SD) approximates well models using IPD. Use of aggregate data will facilitate broader integration of predictive with decision modeling toward the goals of individualized decision-making.

Improving methods of clinical practice guidelines: From guidelines to pathways to fast-and-frugal trees and decision analysis to develop individualised patient care.

BACKGROUND: Current methods for developing clinical practice guidelines have several limitations: they are characterised by the "black box" operation-a process with defined inputs and outputs but an incomplete understanding of its internal workings; they have "the integration problem"-a lack of framework for explicitly integrating factors such as patient preferences and trade-offs between benefits and harms; they generate one recommendation at a time that typically are not connected in a coherent analytical framework; and they apply to "average" patients, while clinicians and their patients seek advice tailored to individual circumstances. METHODS: We propose augmenting the current guideline development method by converting evidence-based pathways into fast-and-frugal decision trees (FFTs) and integrating them with generalised decision curve analysis to formulate clear, individualised management recommendations. RESULTS: We illustrate the process by developing recommendations for the management of heparin-induced thrombocytopenia (HIT). We converted evidence-based pathways for HIT, developed by the American Society of Hematology, into an FFT. Here, we consider only thrombotic complications and major bleeding. We leveraged the predictive potential of FFTs to compare the effects of argatroban, bivalirudin, fondaparinux, and direct oral anticoagulants (DOACs) using generalised decision curve analysis. We found that DOACs were superior to other treatments if the FFT-predicted probability of HIT exceeded 3%. CONCLUSIONS: The proposed analytical framework connects guidelines, pathways, FFTs, and decision analysis, offering risk-tailored personalised recommendations and addressing current guideline development critiques.

Making Decisions When No Further Diagnostic Testing is Available.

In this chapter, we illustrate how evidence about treatments' benefits and harms can be integrated to enable rational decision-making even under considerable clinical uncertainty.

Evidence and Decision-Making.

Today, every country struggles to provide adequate health care to its citizens. Globally, an average of $8.3 trillion or 10% of gross domestic product (GDP) is annually spent on health services. In 2019, the USA spent nearly 18% ($3.2 trillion) of its GDP on health care, projected to reach $6.2 trillion by 2028.

Making Decisions When no Further Diagnostic Testing is Available (Expected Regret Theory Threshold Model).

In Chap. 2 , we illustrated the application of the expected utility theory (EUT) to rational decision-making when no further diagnostic testing is available. In this chapter, we apply regret theory to the decision problems discussed in Chap. 2 .

Decision-Making When Diagnostic Testing is Available.

When a decision-maker has the option of diagnostic testing, they face a typical dilemma: (1) do not administer treatment and do not test, (2) test and decide to administer treatment based on the test result, and (3) administer treatment without testing. In this chapter, we will discuss the theory behind threshold modeling when diagnostic testing is available; we will illustrate the approach by presenting a case vignette.

Formulating Management Strategies Using Fast-and-Frugal Trees (A Decision Tool to Transform Clinical Practice Guidelines and Clinical Pathways into Decision Support at the Point of Care).

Clinical management is rarely based on the collection of one data item. Instead, it is typically characterized by the continuous collection and evaluation of clinical data (symptoms, signs, laboratory, imaging tests, etc.) to establish a platform for further management decisions.

Using Decision Curve Analysis to Evaluate Testing and/or Predictive Modeling.

In this chapter, we extend the threshold model to evaluate the value of diagnostic tests or predictive models over a range of all possible thresholds by using decision curve analysis (DCA). DCA has been developed within the expected utility theory (EUT) and expected regret theory (ERT) framework.

Hybrid and Dual-Processing Threshold Decision Models.

In the previous chapters, we presented various derivations of the threshold model based on the same disease outcomes. We assumed that a decision-maker would calculate the threshold based on either mortality or morbidity outcomes. Basinga and van den Ende derived the threshold by combining both mortality and morbidity outcomes.

Medical Decision-Making and Artificial Intelligence.

In this chapter, we discuss the potential role that artificial intelligence (AI) may have in medical decision-making, the pros and cons, and the limitations and biases that might be introduced when using these novel techniques. As computing becomes more powerful and models continue to grow increasingly more complex, the potential of AI to improve decision-making is increasingly promising. Within many medical fields, however, at the time of this writing (September 2023), the promise of AI is yet to translate into everyday reality. Here, we summarize the role of AI in medical decision-making (diagnosis, prognosis, and treatment).

Which Threshold Model?

As outlined in the Preface (and Chap. 1 and other chapters), this book espoused two fundamental views. The first view consists of the proposal that the threshold model represents a method to address the Sorites paradox, which is a consequence of a relationship between scientific evidence (that exists on a continuum of credibility) and decision-making (that is categorical, yes/no exercises).

Generalised decision curve analysis for explicit comparison of treatment effects.

RATIONALE: Decision curve analysis (DCA) helps integrate prediction models with treatment assessments to guide personalised therapeutic choices among multiple treatment options. However, the current versions of DCA do not explicitly model treatment effects in the analysis but implicitly or holistically assess therapeutic benefits and harms. In addition, the existing DCA cannot allow the comparison of multiple treatments using a standard metric. AIMS AND OBJECTIVES: To develop a generalised version of DCA (gDCA) by decomposing holistically assessed net benefits and harms into patient preferences versus empirical evidence (as obtained in the trials, meta-analyses of clinical studies, etc.) to allow individualised comparison of single or multiple treatments using a common metric. METHODS: We reformulated DCA by (1) decomposing holistic, implicit utilities into specific utilities related to treatment effects and patient's relative values (RV) about disease outcomes versus treatment harms, (2) explicitly modelling each treatment effect at the level of probabilities and/or utilities (outcomes) in a decision tree, and (3) avoiding scaling effects employed in the original DCA to enable comparison of treatment effects against the common metrics. We used data from a published network meta-analysis of randomised trials to inform the use of statin treatment according to Framingham Risk Model. RESULTS: We illustrate the analysis by modelling the effects of three statins in the primary prevention of cardiovascular disease. We performed simultaneous comparisons against standard metrics (RV) for all treatments. We examined for which RV values, a predictive model for guiding personalised treatment, outperformed the strategies of treating everyone or treating no one. We found that the magnitude of benefits (efficacy) seems more important than the simple ratio of efficacy/harms. CONCLUSION: We describe gDCA for evaluating single or multiple treatments to help tailor therapy toward individual risk characteristics. gDCA further helps integrate the principles of evidence-based medicine with decision analysis.

Increasing FDA-accelerated approval of single-arm trials in oncology (1992 to 2020).

OBJECTIVES: We aimed to map the characteristics of single-arm trials (SAT), report the Food and Drug Administration (FDA) transparency in presenting historical control, and to assess the confirmatory randomized controlled trials (RCTs). STUDY DESIGN AND SETTING: This metaresearch included a review of all oncology indication approved using SAT by FDA-AA (FDA-Accelerated Approval) from 1992 to 2020. Two independent reviewers identified SAT, extracted data from FDA full medical reviews for historical controls reported and MEDLINE for searching for confirmatory RCT published. RESULTS: Of 254 FDA-AA approvals, 119 (47%) were approved for oncologic indications using SAT. Fifty-four drugs for 72 oncology indications were for leukemia, lymphoma, lung cancer, urothelial cancer, multiple myeloma, and thyroid cancer. Overall, 37 (52%) treatments were converted into regular approval. Of these, 17 (46%) were based on confirmatory RCTs using overall survival (OS) as an outcome. Five indications were withdrawn from the market. Most trials outcomes were blindly assessed by independent research committees. Median trial sample size was 105 patients (min:8 to max:532). The FDA did not fully specify historical control selection in 75% of cases. CONCLUSION: The granting of FDA-AAs based on SAT in oncology is increasing with more target drugs approved over time. Transparency in historical control reporting is necessary.

Regret affects the choice between neoadjuvant therapy and upfront surgery for potentially resectable pancreatic cancer.

BACKGROUND: When treating potentially resectable pancreatic adenocarcinoma, therapeutic decisions are left to the sensibility of treating clinicians who, faced with a decision that post hoc can be proven wrong, may feel a sense of regret that they want to avoid. A regret-based decision model was applied to evaluate attitudes toward neoadjuvant therapy versus upfront surgery for potentially resectable pancreatic adenocarcinoma. METHODS: Three clinical scenarios describing high-, intermediate-, and low-risk disease-specific mortality after upfront surgery were presented to 60 respondents (20 oncologists, 20 gastroenterologists, and 20 surgeons). Respondents were asked to report their regret of omission and commission regarding neoadjuvant chemotherapy on a scale between 0 (no regret) and 100 (maximum regret). The threshold model and a multilevel mixed regression were applied to analyze respondents' attitudes toward neoadjuvant therapy. RESULTS: The lowest regret of omission was elicited in the low-risk scenario, and the highest regret in the high-risk scenario (P < .001). The regret of the commission was diametrically opposite to the regret of omission (P ≤ .001). The disease-specific threshold mortality at which upfront surgery is favored over the neoadjuvant therapy progressively decreased from the low-risk to the high-risk scenarios (P ≤ .001). The nonsurgeons working in or with lower surgical volume centers (P = .010) and surgeons (P = .018) accepted higher disease-specific mortality after upfront surgery, which resulted in the lower likelihood of adopting neoadjuvant therapy. CONCLUSION: Regret drives decision making in the management of pancreatic adenocarcinoma. Being a surgeon or a specialist working in surgical centers with lower patient volumes reduces the likelihood of recommending neoadjuvant therapy.

Decomposing clinical practice guidelines panels' deliberation into decision theoretical constructs.

RATIONALE, AIMS AND OBJECTIVES: The development of clinical practice guidelines (CPG) suffers from the lack of an explicit and transparent framework for synthesising the key elements necessary to formulate practice recommendations. We matched deliberations of the American Society of Haematology (ASH) CPG panel for the management of pulmonary embolism (PE) with the corresponding decision-theoretical constructs to assess agreement of the panel recommendations with explicit decision modelling. METHODS: Five constructs were identified of which three were used to reformulate the panel's recommendations: (1) standard, expected utility threshold (EUT) decision model; (2) acceptable regret threshold model (ARg) to determine the frequency of tolerable false negative (FN) or false positive (FP) recommendations, and (3) fast-and-frugal tree (FFT) decision trees to formulate the entire strategy for management of PE. We compared four management strategies: withhold testing versus d-dimer → computerized pulmonary angiography (CTPA) ('ASH-Low') versus CTPA→ d-dimer ('ASH-High') versus treat without testing. RESULTS: Different models generated different recommendations. For example, according to EUT, testing should be withheld for prior probability PE < 0.13%, a clinically untenable threshold which is up to 15 times (2/0.13) below the ASH guidelines threshold of ruling out PE (at post probability of PE ≤ 2%). Three models only agreed that the 'ASH low' strategy should be used for the range of pretest probabilities of PE between 0.13% and 13.27% and that the 'ASH high' management should be employed in a narrow range of the prior PE probabilities between 90.85% and 93.07%. For all other prior probabilities of PE, choosing one model did not ensure coherence with other models. CONCLUSIONS: CPG panels rely on various decision-theoretical strategies to develop its recommendations. Decomposing CPG panels' deliberation can provide insights if the panels' deliberation retains a necessary coherence in developing guidelines. CPG recommendations often do not agree with the EUT decision analysis, widely used in medical decision-making modelling.