RESUMEN
OBJECTIVE: To map whether and how systematic reviews (SRs) with network meta-analysis (NMA) use presentation formats to report (a) structured evidence summaries - here defined as reporting of effects estimates in absolute effects with certainty ratings and with a method to rate interventions across one or more outcome(s) - and (b) NMA results in general. STUDY DESIGN AND SETTING: We conducted a systematic survey, searching MEDLINE (Ovid) for SRs with NMA published between January 1, 2020, and December 31, 2021. We planned to include a random sample of publications, with predefined mechanisms in place for saturation, and included SRs that met pre-specified quality criteria and extracted data on presentation formats that reported: (a) estimates of effects, (b) certainty of the evidence, or (c) rating of interventions. RESULTS: The 200 eligible SRs, from 158 unique Journals, utilized 1133 presentation formats. We found structured evidence summaries in 10 publications (5.0%), with three (1.5%) reporting structured evidence summaries across all outcomes, including benefits and harms. Sixteen of the 133 SRs (11.7%) reporting dichotomous outcomes included estimates of absolute effects. Seventy-six SRs (38.0%) reported both benefits and harms and 26 SRs (13.0%) reported certainty ratings in presentation formats, 20 (76.9%) used Grading of Recommendations Assessment, Development and Evaluation (GRADE) and six (23.1%) used Confidence In Network Meta-analysis (CINeMA). Surface Under the Cumulative Ranking Curve (SUCRA) was the most common method to rate interventions (69 SRs, 34.5%). NMA results were most often reported using forest plots (108 SRs, 54.0%) and league tables (93 SRs, 46.5%). CONCLUSION: Most SRs with NMA do not report structured evidence summaries and only rarely do such summaries include reporting of both benefits and harms. Those that do offer effective user-friendly communication and provide models for optimal NMA presentation practice.
RESUMEN
OBJECTIVES: To evaluate and improve "Making Alternative Treatment Choices Intuitive and Trustworthy" (MATCH-IT)-a digital, interactive decision support tool displaying structured evidence summaries for multiple comparisons-to help physicians interpret and apply evidence from network meta-analysis (NMA) for their clinical decision-making. STUDY DESIGN AND SETTING: We conducted a qualitative user testing study, applying principles from user-centered design in an iterative development process. We recruited a convenience sample of practicing physicians in Norway, Belgium, and Canada, and asked them to interpret structured evidence summaries for multiple comparisons-linked to clinical guideline recommendations-displayed in MATCH-IT. User testing included (a) introduction of a clinical scenario, (b) a think-aloud session with participant-tool interaction, and (c) a semistructured interview. We video recorded, transcribed, and analyzed user tests using directed content analysis. The results informed new updates in MATCH-IT. RESULTS: Distributed across 5 development cycles we tested MATCH-IT with 26 physicians. Of these, 24 (94%) reported either no or sparse prior experience with interpretation of NMA. Physicians perceived MATCH-IT as easy to interpret and navigate, and appreciated its ability to provide an overview of the evidence. Visualization of effects in pictograms and inclusion of information on burden of treatment ("practical issues") were highlighted as potentially useful features in interacting with patients. We also identified problems, including undiscovered functionalities (drag and drop), suboptimal tutorial, and cumbersome navigation of the tool. In addition, physicians wanted definition/explanation of key terms (eg, outcomes and "certainty"), and there were concerns that overwhelming evidence from a large NMA would complicate applicability to clinical practice. This led to several updates with development of a new start page, tutorial, updated user interface for more efficient maneuvering, solutions to display definition of key terms and a "frequently asked questions" section. To facilitate interpretation of large networks, we improved categorization of results using color coding and added filtering functionality. These modifications allowed physicians to focus on interventions of interest and reduce information overload. CONCLUSION: This study provides proof of concept that physicians can use MATCH-IT to understand NMA evidence. Key features of MATCH-IT in a clinical context include providing an overview of the evidence, visualization of effects, and the display of information on burden of treatments. However, unfamiliarity with the Grading of Recommendations Assessment, Development and Evaluation concepts, time constraints, and accessibility at the point of care may be challenges for use. To what extent our results are transferable to real-world clinical contexts remains to be explored.
RESUMEN
OBJECTIVES: To investigate medical students' ability to interpret evidence, as well as their self-assessed understandability, perceived usefulness and preferences for design alternatives in an interactive decision support tool, displaying GRADE evidence summaries for multiple treatment options (Making Alternative Treatment CHoices Intuitive and Trustworthy, MATCH-IT). DESIGN: A combined randomised controlled trial and survey. Participants were presented with a clinical scenario and randomised to one of two versions of the MATCH-IT tool (A/B), instructed to explore the evidence and decide on a recommendation. Participants answered a questionnaire assessing interpretation, treatment recommendation self-assessed understandability and perceived usefulness before exposure to the other MATCH-IT version and asked questions on design preferences. SETTING: Online lecture in an evidence-based medicine (EBM) introductory course. PARTICIPANTS: 149 third-year medical students. 52% (n=77) had 6 months of clinical training and 48% (n=72) had preclinical training only. INTERVENTIONS: The MATCH-IT tool version A uses colour coding to categorise interventions by magnitude and direction of effects and displays all outcomes in a table on entry. Version B has no colour coding, and the user must decide which outcomes to display in the table. MAIN OUTCOME MEASURES: Interpretation of evidence, treatment recommendation, perceived usefulness and understandability, preference for format and design alternatives. RESULTS: 82.5% (n=123) of medical students correctly answered ≥4 out of 5 multiple choice questions assessing interpretation of data. 75.8% (n=114) of students made a treatment recommendation in accordance with an expert panel for the same clinical scenario. 87.2% (n=130) found the tool understandable while 91.9% perceived the tool as useful in addressing the clinical scenario. CONCLUSION: Medical students with no prior training in EBM can interpret and use the MATCH-IT tool. Certain design alternatives were preferred but had no bearing on interpretation of evidence or understandability of the tool.