RESUMEN
OBJECTIVES: To evaluate and improve "Making Alternative Treatment Choices Intuitive and Trustworthy" (MATCH-IT)-a digital, interactive decision support tool displaying structured evidence summaries for multiple comparisons-to help physicians interpret and apply evidence from network meta-analysis (NMA) for their clinical decision-making. STUDY DESIGN AND SETTING: We conducted a qualitative user testing study, applying principles from user-centered design in an iterative development process. We recruited a convenience sample of practicing physicians in Norway, Belgium, and Canada, and asked them to interpret structured evidence summaries for multiple comparisons-linked to clinical guideline recommendations-displayed in MATCH-IT. User testing included (a) introduction of a clinical scenario, (b) a think-aloud session with participant-tool interaction, and (c) a semistructured interview. We video recorded, transcribed, and analyzed user tests using directed content analysis. The results informed new updates in MATCH-IT. RESULTS: Distributed across 5 development cycles we tested MATCH-IT with 26 physicians. Of these, 24 (94%) reported either no or sparse prior experience with interpretation of NMA. Physicians perceived MATCH-IT as easy to interpret and navigate, and appreciated its ability to provide an overview of the evidence. Visualization of effects in pictograms and inclusion of information on burden of treatment ("practical issues") were highlighted as potentially useful features in interacting with patients. We also identified problems, including undiscovered functionalities (drag and drop), suboptimal tutorial, and cumbersome navigation of the tool. In addition, physicians wanted definition/explanation of key terms (eg, outcomes and "certainty"), and there were concerns that overwhelming evidence from a large NMA would complicate applicability to clinical practice. This led to several updates with development of a new start page, tutorial, updated user interface for more efficient maneuvering, solutions to display definition of key terms and a "frequently asked questions" section. To facilitate interpretation of large networks, we improved categorization of results using color coding and added filtering functionality. These modifications allowed physicians to focus on interventions of interest and reduce information overload. CONCLUSION: This study provides proof of concept that physicians can use MATCH-IT to understand NMA evidence. Key features of MATCH-IT in a clinical context include providing an overview of the evidence, visualization of effects, and the display of information on burden of treatments. However, unfamiliarity with the Grading of Recommendations Assessment, Development and Evaluation concepts, time constraints, and accessibility at the point of care may be challenges for use. To what extent our results are transferable to real-world clinical contexts remains to be explored.
RESUMEN
BACKGROUND: Tools for shared decision-making (e.g. decision aids) are intended to support health care professionals and patients engaged in clinical encounters involving shared decision-making. However, decision aids are hard to produce, and onerous to update. Consequently, they often do not reflect best current evidence, and show limited uptake in practice. In response, we initiated the Sharing Evidence to Inform Treatment decisions (SHARE-IT) project. Our goal was to develop and refine a new generation of decision aids that are generically produced along digitally structured guidelines and evidence summaries. METHODS: Applying principles of human-centred design and following the International Patient Decision Aid Standards (IPDAS) and GRADE methods for trustworthy evidence summaries we developed a decision aid prototype in collaboration with the Developing and Evaluating Communication strategies to support Informed Decisions and practice based on Evidence project (DECIDE). We iteratively user-tested the prototype in clinical consultations between clinicians and patients. Semi-structured interviews of participating clinicians and patients were conducted. Qualitative content analysis of both user-testing sessions and interviews was performed and results categorized according to a revised Morville's framework of user-experience. We made it possible to produce, publish and use these decision aids in an electronic guideline authoring and publication platform (MAGICapp). RESULTS: Direct observations and analysis of user-testing of 28 clinical consultations between physicians and patients informed four major iterations that addressed readability, understandability, usability and ways to cope with information overload. Participants reported that the tool supported natural flow of the conversation and induced a positive shift in consultation habits towards shared decision-making. We integrated the functionality of SHARE-IT decision aids in MAGICapp, which has since generated numerous decision aids. CONCLUSION: Our study provides a proof of concept that encounter decision aids can be generically produced from GRADE evidence summaries and clinical guidelines. Online authoring and publication platforms can help scale up production including continuous updating of electronic encounter decision aids, fully integrated with evidence summaries and clinical practice guidelines.