Integrating Explainable Machine Learning in Clinical Decision Support Systems: Study Involving a Modified Design Thinking Approach.

BACKGROUND: Though there has been considerable effort to implement machine learning (ML) methods for health care, clinical implementation has lagged. Incorporating explainable machine learning (XML) methods through the development of a decision support tool using a design thinking approach is expected to lead to greater uptake of such tools. OBJECTIVE: This work aimed to explore how constant engagement of clinician end users can address the lack of adoption of ML tools in clinical contexts due to their lack of transparency and address challenges related to presenting explainability in a decision support interface. METHODS: We used a design thinking approach augmented with additional theoretical frameworks to provide more robust approaches to different phases of design. In particular, in the problem definition phase, we incorporated the nonadoption, abandonment, scale-up, spread, and sustainability of technology in health care (NASSS) framework to assess these aspects in a health care network. This process helped focus on the development of a prognostic tool that predicted the likelihood of admission to an intensive care ward based on disease severity in chest x-ray images. In the ideate, prototype, and test phases, we incorporated a metric framework to assess physician trust in artificial intelligence (AI) tools. This allowed us to compare physicians' assessments of the domain representation, action ability, and consistency of the tool. RESULTS: Physicians found the design of the prototype elegant, and domain appropriate representation of data was displayed in the tool. They appreciated the simplified explainability overlay, which only displayed the most predictive patches that cumulatively explained 90% of the final admission risk score. Finally, in terms of consistency, physicians unanimously appreciated the capacity to compare multiple x-ray images in the same view. They also appreciated the ability to toggle the explainability overlay so that both options made it easier for them to assess how consistently the tool was identifying elements of the x-ray image they felt would contribute to overall disease severity. CONCLUSIONS: The adopted approach is situated in an evolving space concerned with incorporating XML or AI technologies into health care software. We addressed the alignment of AI as it relates to clinician trust, describing an approach to wire framing and prototyping, which incorporates the use of a theoretical framework for trust in the design process itself. Moreover, we proposed that alignment of AI is dependent upon integration of end users throughout the larger design process. Our work shows the importance and value of engaging end users prior to tool development. We believe that the described approach is a unique and valuable contribution that outlines a direction for ML experts, user experience designers, and clinician end users on how to collaborate in the creation of trustworthy and usable XML-based clinical decision support tools.

Patient-oriented evidence that matters (POEMs)™ suggest potential clinical topics for the Choosing Wisely™ campaign.

OBJECTIVE: We propose a method of identifying clinical topics for campaigns like Choosing Wisely. METHODS: In the context of an ongoing continuing medication education program, we analyzed ratings on every patient-oriented evidence that matters (POEM) synopsis delivered in 2012 and 2013. Given the objective of the Choosing Wisely campaign, we focused this analysis on 1 specific item in the validated questionnaire used by physicians to rate POEMs. This questionnaire item is about "avoiding an unnecessary diagnostic test or treatment." For each POEM, we calculated frequencies and proportions for this item, then we identified the 20 POEMs that were most commonly associated with this item in 2012 and 2013. Finally, we determined whether the clinical topic of each of these POEMs was mentioned in the Choosing Wisely master list. RESULTS: In 2012 and 2013 we received 506,809 completed questionnaires (or ratings) linked to 530 POEMs, for an average of 956 ratings per POEM. In 59% of these POEMs (n = 312), the most commonly expected type of health benefit was "avoiding an unnecessary diagnostic test or treatment." We then identified the top 20 POEMs most commonly associated with this item in each year by ranking all 312 POEMs from the top down. The clinical topic addressed by 29 of these 40 POEMs was not addressed in the Choosing Wisely master list. These topics fell into 3 categories: diagnostic tests, medical interventions, and surgical interventions. CONCLUSION: "Big data" can identify clinical topics relevant to campaigns such as Choosing Wisely. This process represents a new way to inform the expert panel approach.

Physician assessments of the value of therapeutic information delivered via e-mail.

PROBLEM ADDRESSED: Although e-learning programs are popular and access to electronic knowledge resources has improved, raising awareness about updated therapeutic recommendations in practice continues to be a challenge. OBJECTIVE OF PROGRAM: To raise awareness about and document the use of therapeutic recommendations. PROGRAM DESCRIPTION: In 2010, family physicians evaluated e-Therapeutics (e-T) Highlights with a Web-based tool called the Information Assessment Method (IAM). The e-T Highlights consisted of information found in the primary care reference e-Therapeutics+. Each week, family physicians received an e-mail containing a link to 1 Highlight from a different chapter of e-Therapeutics+. Family physicians received continuing medical education credits for each Highlight they rated with the IAM. Of the 5346 participants, 85% of them were full-time or part-time practitioners. A total of 31 429 Highlights ratings were received in 2010 (median of 2 ratings per participant, range 1 to 49). Among participants who rated more than 2 Highlights, the median number of ratings was 7 (mean 11.9). The relevance of the information from individual Highlights varied widely; however, for 90% of the rated Highlights participants indicated total or partial relevance of the information for at least 1 patient. For 41% of rated Highlights, participants expected patient health benefits to result from implementing the recommendation, such as avoiding an unnecessary or inappropriate treatment, or a preventive intervention. CONCLUSION: This continuing medical education program stimulated family physicians to rate therapeutic recommendations that were delivered weekly via e-mail. The process of rating e-T Highlights with the IAM raised awareness about treatment recommendations and documented self-reported use of this information in practice.

Development and content validation of the information assessment method for patients and consumers.

BACKGROUND: Online consumer health information addresses health problems, self-care, disease prevention, and health care services and is intended for the general public. Using this information, people can improve their knowledge, participation in health decision-making, and health. However, there are no comprehensive instruments to evaluate the value of health information from a consumer perspective. OBJECTIVE: We collaborated with information providers to develop and validate the Information Assessment Method for all (IAM4all) that can be used to collect feedback from information consumers (including patients), and to enable a two-way knowledge translation between information providers and consumers. METHODS: Content validation steps were followed to develop the IAM4all questionnaire. The first version was based on a theoretical framework from information science, a critical literature review and prior work. Then, 16 laypersons were interviewed on their experience with online health information and specifically their impression of the IAM4all questionnaire. Based on the summaries and interpretations of interviews, questionnaire items were revised, added, and excluded, thus creating the second version of the questionnaire. Subsequently, a panel of 12 information specialists and 8 health researchers participated in an online survey to rate each questionnaire item for relevance, clarity, representativeness, and specificity. The result of this expert panel contributed to the third, current, version of the questionnaire. RESULTS: The current version of the IAM4all questionnaire is structured by four levels of outcomes of information seeking/receiving: situational relevance, cognitive impact, information use, and health benefits. Following the interviews and the expert panel survey, 9 questionnaire items were confirmed as relevant, clear, representative, and specific. To improve readability and accessibility for users with a lower level of literacy, 19 items were reworded and all inconsistencies in using a passive or active voice have been solved. One item was removed due to redundancy. The current version of the IAM4all questionnaire contains 28 items. CONCLUSIONS: We developed and content validated the IAM4all in partnership with information providers, information specialists, researchers and representatives of information consumers. This questionnaire can be integrated within electronic knowledge resources to stimulate users' reflection (eg, their intention to use information). We claim that any organization (eg, publishers, community organizations, or patient associations), can evaluate and improve their online consumer health information from a consumers' perspective using this method.

Number needed to benefit from information (NNBI): proposal from a mixed methods research study with practicing family physicians.

PURPOSE: We wanted to describe family physicians' use of information from an electronic knowledge resource for answering clinical questions, and their perception of subsequent patient health outcomes; and to estimate the number needed to benefit from information (NNBI), defined as the number of patients for whom clinical information was retrieved for 1 to benefit. METHODS: We undertook a mixed methods research study, combining quantitative longitudinal and qualitative research studies. Participants were 41 family physicians from primary care clinics across Canada. Physicians were given access to 1 electronic knowledge resource on handheld computer in 2008-2009. For the outcome assessment, participants rated their searches using a validated method. Rated searches were examined during interviews guided by log reports that included ratings. Cases were defined as clearly described searches where clinical information was used for a specific patient. For each case, interviewees described information-related patient health outcomes. For the mixed methods data analysis, quantitative and qualitative data were merged into clinical vignettes (each vignette describing a case). We then estimated the NNBI. RESULTS: In 715 of 1,193 searches for information conducted during an average of 86 days, the search objective was directly linked to a patient. Of those searches, 188 were considered to be cases. In 53 cases, participants associated the use of information with at least 1 patient health benefit. This finding suggested an NNBI of 14 (715/53). CONCLUSION: The NNBI may be used in further experimental research to compare electronic knowledge resources. A low NNBI can encourage clinicians to search for information more frequently. If all searches had benefits, the NNBI would be 1. In addition to patient benefits, learning and knowledge reinforcement outcomes are frequently reported.

Do family physicians retrieve synopses of clinical research previously read as email alerts?

BACKGROUND: A synopsis of new clinical research highlights important aspects of one study in a brief structured format. When delivered as email alerts, synopses enable clinicians to become aware of new developments relevant for practice. Once read, a synopsis can become a known item of clinical information. In time-pressured situations, remembering a known item may facilitate information retrieval by the clinician. However, exactly how synopses first delivered as email alerts influence retrieval at some later time is not known. OBJECTIVES: We examined searches for clinical information in which a synopsis previously read as an email alert was retrieved (defined as a dyad). Our study objectives were to (1) examine whether family physicians retrieved synopses they previously read as email alerts and then to (2) explore whether family physicians purposefully retrieved these synopses. METHODS: We conducted a mixed-methods study in which a qualitative multiple case study explored the retrieval of email alerts within a prospective longitudinal cohort of practicing family physicians. Reading of research-based synopses was tracked in two contexts: (1) push, meaning to read on email and (2) pull, meaning to read after retrieval from one electronic knowledge resource. Dyads, defined as synopses first read as email alerts and subsequently retrieved in a search of a knowledge resource, were prospectively identified. Participants were interviewed about all of their dyads. Outcomes were the total number of dyads and their type. RESULTS: Over a period of 341 days, 194 unique synopses delivered to 41 participants resulted in 4937 synopsis readings. In all, 1205 synopses were retrieved over an average of 320 days. Of the 1205 retrieved synopses, 21 (1.7%) were dyads made by 17 family physicians. Of the 1205 retrieved synopses, 6 (0.5%) were known item type dyads. However, dyads also occurred serendipitously. CONCLUSION: In the single knowledge resource we studied, email alerts containing research-based synopses were rarely retrieved. Our findings help us to better understand the effect of push on pull and to improve the integration of research-based information within electronic resources for clinicians.

A participatory approach to understanding and measuring patient satisfaction in a primary care teaching setting.

BACKGROUND: Patient satisfaction is a complex, multidimensional concept that is difficult to measure. However, there is agreement that understanding the expectations of a patient community or "what is important to them" is an essential consideration. We chose a participatory approach to address patient satisfaction in the context of a primary care teaching clinic. OBJECTIVES: The objectives of this project were to use a participatory research team of patients staff and researchers to (1) adapt an existing patient satisfaction questionnaire (PSQ) to the specific cultural and organizational elements ofa primary care teaching clinic, (2) administer the revised questionnaire and use the findings as a tool for organizational improvement, with the ultimate goal of increasing patient satisfaction, and (3) ensure that all decision making involved patients and staff to empower them in the process of organizational change. METHODS: We used an iterative, mixed methods approach to conduct this project. An interdisciplinary committee composed of members of the patient community, clinical and administrative staff, and researchers worked together as the primary decision making body. RESULTS: We modified a preexisting questionnaire to address the unique care delivery model of the clinic, issues of cultural sensitivity, and the need for simplified language and response format. Patient dissatisfaction was found to center on continuity and access to care. CONCLUSIONS: The participatory approach was critical to our success in understanding and measuring patient satisfaction from the patients' perspective. The involvement of the interdisciplinary committee and the high level of joint decision making in this project represents a unique contribution to assessing primary care patient satisfaction.

Impact of research-based synopses delivered as daily e-mail: a prospective observational study.

We conducted a prospective observational study to (1) determine usage and construct validity of a method to gauge the cognitive impact of information derived from daily e-mail, and (2) describe self-reported impacts of research-based synopses (InfoPOEMs) delivered as e-mail. Ratings of InfoPOEMs using an Impact assessment scale provided (a) data on usage of the impact assessment method, (b) reports of impact by InfoPOEM and by doctor and (c) data for analysis of construct validity of the scale. PARTICIPANTS were family physicians or general practitioners who rated at least five InfoPOEMs delivered on e-mail. For each InfoPOEM rated, 0.1 continuing education credit was awarded by the College of Family Physicians of Canada. Use of the impact assessment scale linked to a daily InfoPOEM was sustained during the 150-day study period. 1,007 participants submitted 61,493 reports of 'cognitive impact' by rating on average 61 InfoPOEMs (range 5-111). 'I learned something new' was most frequently reported. 'I was frustrated as there was not enough information or nothing useful' was the most frequently reported negative type of impact. The proportion of reports of 'No Impact' varied substantially across individual InfoPOEMs. Impact patterns suggested an 8 or 9-factor solution. Our Impact assessment method facilitates knowledge transfer by promoting two-way exchange between providers of health information and family doctors. Providers of health information can use this method to better understand the impact of research-based synopses. Sustaining current practice and increasing knowledge about new developments in medicine are important outcomes arising from research-based synopses delivered as e-mail, in addition to practice change.