Toward a responsible future: recommendations for AI-enabled clinical decision support.

BACKGROUND: Integrating artificial intelligence (AI) in healthcare settings has the potential to benefit clinical decision-making. Addressing challenges such as ensuring trustworthiness, mitigating bias, and maintaining safety is paramount. The lack of established methodologies for pre- and post-deployment evaluation of AI tools regarding crucial attributes such as transparency, performance monitoring, and adverse event reporting makes this situation challenging. OBJECTIVES: This paper aims to make practical suggestions for creating methods, rules, and guidelines to ensure that the development, testing, supervision, and use of AI in clinical decision support (CDS) systems are done well and safely for patients. MATERIALS AND METHODS: In May 2023, the Division of Clinical Informatics at Beth Israel Deaconess Medical Center and the American Medical Informatics Association co-sponsored a working group on AI in healthcare. In August 2023, there were 4 webinars on AI topics and a 2-day workshop in September 2023 for consensus-building. The event included over 200 industry stakeholders, including clinicians, software developers, academics, ethicists, attorneys, government policy experts, scientists, and patients. The goal was to identify challenges associated with the trusted use of AI-enabled CDS in medical practice. Key issues were identified, and solutions were proposed through qualitative analysis and a 4-month iterative consensus process. RESULTS: Our work culminated in several key recommendations: (1) building safe and trustworthy systems; (2) developing validation, verification, and certification processes for AI-CDS systems; (3) providing a means of safety monitoring and reporting at the national level; and (4) ensuring that appropriate documentation and end-user training are provided. DISCUSSION: AI-enabled Clinical Decision Support (AI-CDS) systems promise to revolutionize healthcare decision-making, necessitating a comprehensive framework for their development, implementation, and regulation that emphasizes trustworthiness, transparency, and safety. This framework encompasses various aspects including model training, explainability, validation, certification, monitoring, and continuous evaluation, while also addressing challenges such as data privacy, fairness, and the need for regulatory oversight to ensure responsible integration of AI into clinical workflow. CONCLUSIONS: Achieving responsible AI-CDS systems requires a collective effort from many healthcare stakeholders. This involves implementing robust safety, monitoring, and transparency measures while fostering innovation. Future steps include testing and piloting proposed trust mechanisms, such as safety reporting protocols, and establishing best practice guidelines.

Enhancing Access Through Language-Tailored Approach in Telehealth and Veterans Video Connect: Traumatic Brain Injury (TBI) Veterans Satisfaction Assessment.

INTRODUCTION: Traumatic brain injury (TBI) can impact language processing, necessitating language-tailored approaches. Telehealth may expand rural Veterans' access but has unknown feasibility for language preferences. This study explored telehealth/Veterans Video Connect satisfaction for Spanish/English TBI screening. MATERIALS AND METHODS: The study was approved by the VA Caribbean Healthcare System Institutional Review Board and the Research and Development Committee. Mixed methods evaluated telehealth satisfaction in Veterans receiving TBI assessments from October 2021 to October 2023. Surveys included the 16-item Clinical Video Telehealth (CVT) questionnaire on communication, technical factors, coordination, and overall satisfaction, and the 21-item Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) examining usefulness, ease of use, manners, quality, and satisfaction. Mean domain/item scores were calculated among 57 Veterans, 12 English, and 45 Spanish-speaking. Semi-structured interviews also elicited user experiences from 4 providers and 5 Veterans. Transcripts underwent qualitative coding for themes using Atlas.ti.8. RESULTS: On CVT (0-5 scale), overall satisfaction averaged 4.50 (English) and 4.69 (Spanish). Lowest scoring item for English users was easy video connection (4.25), while unclear expectations had the lowest Spanish score (3.60). For TSUQ, overall mean scores were 4.50 (English) and 4.67 (Spanish), with improved health post-telehealth having the lowest average (English 3.33, Spanish 3.67). Qualitatively, Veterans and providers noted strengths like access and communication but weaknesses around connectivity, care delays, and privacy. Differences emerged regarding convenience (Veterans) versus operational barriers (providers). There was a strong positive correlation for Spanish surveys and a moderate correlation for English surveys (r = 0.71 Spanish surveys, r = 0.69 English surveys) between TSUQ and CVT for individual respondents. CONCLUSIONS: Patients conveyed positive experiences, but qualitative data revealed actionable targets for optimization like infrastructure and coordination improvements. Key limitations include small samples and lack of comparison to in-person care. Still, high satisfaction coupled with specific user feedback highlights telehealth's potential while directing enhancements. The results found high Veteran satisfaction with Spanish/English TBI telehealth, but mixed methods illuminated salient domains for better accommodating user needs, particularly regarding logistics and technology. Rigorously integrating experiences with metrics over expanded diverse samples and modalities can further guide refinements to enhance telehealth with a language-tailored approach.

Exploring How Youth Use TikTok for Mental Health Information in British Columbia: Semistructured Interview Study With Youth.

BACKGROUND: TikTok (ByteDance) experienced a surge in popularity during the COVID-19 pandemic as a way for people to interact with others, share experiences and thoughts related to the pandemic, and cope with ongoing mental health challenges. However, few studies have explored how youth use TikTok to learn about mental health. OBJECTIVE: This study aims to understand how youth used TikTok during the COVID-19 pandemic to learn about mental health and mental health support. METHODS: Semistructured interviews were conducted with 21 youths (aged 12-24 years) living in British Columbia, Canada, who had accessed TikTok for mental health information during the COVID-19 pandemic. Interviews were audio-recorded, transcribed verbatim, coded, and analyzed using an inductive, data-driven approach. RESULTS: A total of 3 overarching themes were identified describing youth's experiences. The first theme centered on how TikTok gave youth easy access to mental health information and support, which was particularly helpful during the COVID-19 pandemic to curb the effects of social isolation and the additional challenges of accessing mental health services. The second theme described how the platform provided youth with connection, as it gave youth a safe space to talk about mental health and allowed them to feel seen by others going through similar experiences. This helped normalize and destigmatize conversations about mental health and brought awareness to various mental health conditions. Finally, the last theme focused on how this information led to action, such as trying different coping strategies, discussing mental health with peers and family, accessing mental health services, and advocating for themselves during medical appointments. Across the 3 themes, youth expressed having to be mindful of bias and misinformation, highlighting the barriers to identifying and reporting misinformation and providing individualized advice on the platform. CONCLUSIONS: Findings suggest that TikTok can be a useful tool to increase mental health awareness, reduce stigma, and encourage youth to learn and address their mental health challenges while providing a source of peer connection and support. Simultaneously, TikTok can adversely impact mental health through repetitive exposure to mentally distressing content and misleading diagnosis and treatment information. Regulations against harmful content are needed to mitigate these risks and make TikTok safer for youth. Efforts should also be made to increase media and health literacy among youth so that they can better assess the information they consume online.

A mixed methods analysis of existing assessment and evaluation tools (AETs) for mental health applications.

Introduction: Mental health Applications (MH Apps) can potentially improve access to high-quality mental health care. However, the recent rapid expansion of MH Apps has created growing concern regarding their safety and effectiveness, leading to the development of AETs (Assessment and Evaluation Tools) to help guide users. This article provides a critical, mixed methods analysis of existing AETs for MH Apps by reviewing the criteria used to evaluate MH Apps and assessing their effectiveness as evaluation tools. Methods: To identify relevant AETs, gray and scholarly literature were located through stakeholder consultation, Internet searching via Google and a literature search of bibliographic databases Medline, APA PsycInfo, and LISTA. Materials in English that provided a tool or method to evaluate MH Apps and were published from January 1, 2000, to January 26, 2021 were considered for inclusion. Results: Thirteen relevant AETs targeted for MH Apps met the inclusion criteria. The qualitative analysis of AETs and their evaluation criteria revealed that despite purporting to focus on MH Apps, the included AETs did not contain criteria that made them more specific to MH Apps than general health applications. There appeared to be very little agreed-upon terminology in this field, and the focus of selection criteria in AETs is often IT-related, with a lesser focus on clinical issues, equity, and scientific evidence. The quality of AETs was quantitatively assessed using the AGREE II, a standardized tool for evaluating assessment guidelines. Three out of 13 AETs were deemed 'recommended' using the AGREE II. Discussion: There is a need for further improvements to existing AETs. To realize the full potential of MH Apps and reduce stakeholders' concerns, AETs must be developed within the current laws and governmental health policies, be specific to mental health, be feasible to implement and be supported by rigorous research methodology, medical education, and public awareness.

Identifying the capabilities for creating next-generation registries: a guide for data leaders and a case for "registry science".

OBJECTIVE: The increasing demands for curated, high-quality research data are driving the emergence of a novel registry type. The need to assemble, curate, and export this data grows, and the conventional simplicity of registry models is driving the need for advanced, multimodal data registries-the dawn of the next-generation registry. MATERIALS AND METHODS: The article provides an outline of the technology roles and responsibilities needed for successful implementations of next-generation registries. RESULTS: We propose a framework for the planning, construction, maintenance, and sustainability of this new registry type. DISCUSSION: A rubric of organizational, computational, and human resource needs is discussed in detail, backed by over 40 years of combined in-the-field experiences by the authors. CONCLUSIONS: A novel field, registry science, within the clinical research informatics domain, has arisen to offer its insights into conceiving, structuring, and sustaining this new breed of tools.

Evaluation of the design and structure of electronic medication labels to improve patient health knowledge and safety: a systematic review.

INTRODUCTION: Patient misunderstanding of instructions on medication labels is a common cause of medication errors and can result in ineffective treatment. One way to better improve patient comprehension of medication labels is by optimizing the content and display of the information. OBJECTIVES: To review comparative studies that have evaluated the design of a medication label to improve patient knowledge or safety. METHODS: Studies were selected from systematic computerized literature searches performed in PubMed, Embase (Elsevier), Cochrane Central (EBSCO), Cumulative Index to Nursing and Allied Health Literature-CINAHL (EBSCO), and Web of Science (Thomson Reuters). Eligible studies included comparative studies that evaluated the design of a medication label to improve patient knowledge or safety. RESULTS: Of the 246 articles identified in the primary literature search, 14 studies were selected for data abstraction. Thirteen of these studies significantly impacted the patient understanding of medication labels. Three studies included a measure of patient safety in terms of medication adherence and dosing errors. The utilization of patient-centered language, pictograms/graphics, color/white space, or font optimization was seen to have the most impact on patient comprehension. CONCLUSION: It is essential to present medication information in an optimal manner for patients. This can be done by standardizing the content, display, and format of medication labels to improve understanding and medication usage. Evidence-based design principles can, therefore, be used to facilitate the standardization of the structure of label content for both print and electronic devices. However, more research needs to be done on validating the implications of label content display to measure its impact on patient safety. SYSTEMIC REVIEW REGISTRATION: PROSPERO CRD42022347510 ( http://www.crd.york.ac.uk/prospero/ ).

Stakeholders' views and opinions on existing guidelines on "How to Choose Mental Health Apps".

Background: Mental health Applications (Mhealth Apps) can change how healthcare is delivered. However, very little is known about the efficacy of Mhealth Apps. Currently, only minimum guidance is available in Assessment and Evaluation Tools (AETs). Therefore, this project aims to understand AET developers' perspectives and end users' experiences and opinions on "how to choose a Mhealth App". Objective: The primary objectives were: (1) obtaining stakeholder's opinions and experiences of development and use of AETs for Mhealth Apps, their weaknesses and strengths, and barriers in their implementation of Mhealth Apps; (2) the experiences of App users, their analyzation and, obstacles in the use of apps; and (3) to quantify themes related to choosing a Mhealth App. Methods: This qualitative study, used a sampling method to recruit six stakeholders (one App developer, two AET developers, an individual with lived experience of mental health illness, and two physicians) who were interviewed using a topic guide. These were examined by researchers (CT, WK, & FN) using thematic content analysis. Additionally, an anonymous online survey of 107 individuals was conducted. Findings: Our analyses revealed six main themes: (a) needs and opportunities; (b) views on Mhealth apps; (c) views & opinions on AETs; (d) implementation barriers; (e) system of evaluation and; (f) future directions. The first key concept was, all stakeholders agreed that Apps could significantly impact mental health and that end-users were unaware of mental health AETs and Apps. Secondly, due to commercial interests, end-users reliability of App evaluations requires clear conflict-free guidelines. Thirdly, AETs should be evaluated and developed through a rigorous methodology. Finally, stakeholders shared insights into future developments for AETs and Mhealth Apps. Additionally, online survey respondents chose a "health professional" as their preferred source of guidance in selecting a Mhealth app (84%) and best suited to develop guidelines (70%). Conclusion: The interviews and survey highlight the need for Mhealth Apps to be regulated and the importance of health professionals' engagement in the implementation process. Similarly, without well-defined roles for App evaluations within the health care system, it is unlikely that AETs will have wider spread use and impact without risk.

Applying Logic to the Healthcare Journey.

Cross-disciplinary approaches to remediate complex healthcare service delivery issues may have merit. This study aims to establish the potential benefits of applying service design and evaluative research concepts in healthcare. Specifically, this study aims to demonstrate how a Customer Journey Map and a Logic Model could be used in unison to identify and remedy service delivery gaps to reduce barriers to care. This study provides systems thinking approach to solving operational issues in healthcare.

A Human-Centered Physician Journey.

The COVID-19 pandemic has placed unprecedented strain on global health systems, and the ability to safely and effectively deliver care. Further, it has impacted the mental health of global populations, in particular healthcare providers (i.e., physicians, nurses). In a service delivery context, much can be learned about empathy both from a provider and patient lens. Thus, the literature was explored to see if the concept of journey mapping was used to illustrate the intersections and pain points of the clinical workflow along the physician journey.

Humanizing Big Data and Detailing Social Determinants of Health via Information Visualizations.

The pandemic has had devastating impacts on humanity and the global healthcare sector. An analysis into the social determinants of health, in particular racial and ethnic disparities may explain why certain population groups have been disproportionately affected by COVID-19. The objective of this study is to humanize and personify numerical data. Additionally, COVID-19 population data will be stratified via three data visualization tools (i.e., a persona, a journey map, Sankey diagram) to create a Visualized Combined Experience (VCE) Diagram to illustrate the micro, and macro, perspectives of marginalized individuals across the continuum of care.

Exploring Patient Journey Mapping and the Learning Health System: Scoping Review.

BACKGROUND: Journey maps are visualization tools that can facilitate the diagrammatical representation of stakeholder groups by interest or function for comparative visual analysis. Therefore, journey maps can illustrate intersections and relationships between organizations and consumers using products or services. We propose that some synergies may exist between journey maps and the concept of a learning health system (LHS). The overarching goal of an LHS is to use health care data to inform clinical practice and improve service delivery processes and patient outcomes. OBJECTIVE: The purpose of this review was to assess the literature and establish a relationship between journey mapping techniques and LHSs. Specifically, in this study, we explored the current state of the literature to answer the following research questions: (1) Is there a relationship between journey mapping techniques and an LHS in the literature? (2) Is there a way to integrate the data from journey mapping activities into an LHS? (3) How can the data gleaned from journey map activities be used to inform an LHS? METHODS: A scoping review was conducted by querying the following electronic databases: Cochrane Database of Systematic Reviews (Ovid), IEEE Xplore, PubMed, Web of Science, Academic Search Complete (EBSCOhost), APA PsycInfo (EBSCOhost), CINAHL (EBSCOhost), and MEDLINE (EBSCOhost). Two researchers applied the inclusion criteria and assessed all articles by title and abstract in the first screen, using Covidence. Following this, a full-text review of included articles was done, with relevant data extracted, tabulated, and assessed thematically. RESULTS: The initial search yielded 694 studies. Of those, 179 duplicates were removed. Following this, 515 articles were assessed during the first screening phase, and 412 were excluded, as they did not meet the inclusion criteria. Next, 103 articles were read in full, and 95 were excluded, resulting in a final sample of 8 articles that satisfied the inclusion criteria. The article sample can be subsumed into 2 overarching themes: (1) the need to evolve service delivery models in health care, and (2) the potential value of using patient journey data in an LHS. CONCLUSIONS: This scoping review demonstrated the gap in knowledge regarding integrating the data from journey mapping activities into an LHS. Our findings highlighted the importance of using the data from patient experiences to enrich an LHS and provide holistic care. To satisfy this gap, the authors intend to continue this investigation to establish the relationship between journey mapping and the concept of LHSs. This scoping review will serve as phase 1 of an investigative series. Phase 2 will entail the creation of a holistic framework to guide and streamline data integration from journey mapping activities into an LHS. Lastly, phase 3 will provide a proof of concept to demonstrate how patient journey mapping activities could be integrated into an LHS.

Social Media is Addictive and Influences Behavior: Should it Be Regulated as a Digital Therapeutic?

Recently, we were deeply saddened by the findings of the coroner investigating the death of 14-year-old Molly Russell. Deeply saddened and angry but not surprised. This case should be seen as a sentinel event, given that this is the first time social media was directly implicated as a cause of death. We should use this opportunity to advance proposals for the regulations of the health effects of social media.

Digital vaccine passports and digital health diplomacy: an online model WHO simulation.

In the wake of Coronavirus disease 2019 (COVID-19), several nations have sought to implement digital vaccine passports (DVPs) to enable the resumption of international travel. Comprising a minimum dataset for each unique individual, DVPs have the makings of a global electronic health record, broaching key issues involved in building a global digital health ecosystem. Debate simulations offer a safe, interactive space to foster participatory policy discussions for advancing digital health diplomacy. This study used an online simulation of a Model World Health Assembly to critically analyze the sociotechnical issues associated with the global implementation of DVPs, and to generate useful insights and questions about the role of diplomacy in global digital health. The debate arguments addressed and provided insights into the technological, scientific, ethical, legal, policy, and societal aspects of DVPs. Reflecting on the simulation, we discuss its opportunities and challenges for the digitalization, decolonization, decentralization, and democratization of participatory policymaking.

Implementation and Use of Risk Evaluation and Mitigation Strategies Programs in Practice: A Scoping Review of the Literature.

BACKGROUND: Risk Evaluation and Mitigation Strategy (REMS) programs intend to improve medication safety but can add significant administrative burdens to providers and health systems. Various stakeholders have made efforts to use technology to improve REMS programs. OBJECTIVES: The objective of this scoping review is to describe studies evaluating workflows, automation, or electronic data exchange related to REMS programs. METHODS: A literature search of PubMed, Embase, and Web of Science was performed for articles published between January 2007 and July 2021. Studies were identified using the relevant Medical Subject Headings terms and related keywords. Articles must have described a workflow change and measured the impact of the workflow change. RESULTS: Of 299 citations initially identified, 7 were included in the final review after removing duplicates and articles not meeting predefined inclusion criteria. Included studies consisted of three manuscripts and four conference abstracts. Electronic health record interventions, such as customized order sets and clinical decision support alerts, were the most common strategy reported. Other strategies included developing a portal to verify REMS enrollment, requiring prescribers to contact the dispensing pharmacy, provider education, and restrictions based on the provider specialty. One study evaluated automated data exchange for REMS program processes in a mock environment. Although only three studies reported baseline metrics for comparison, all studies noted an improvement or benefit to the implemented workflow process changes. CONCLUSION: There is limited evidence describing REMS workflows, automation, and electronic data exchange. Various strategies to address REMS program requirements were reported, but no studies described the use of data exchange standards in a real-world setting despite efforts by the Food and Drug Administration and other stakeholders. Additional efforts are needed to automate REMS programs.

Alignment of two standard terminologies for dosage form: RxNorm from the National Library of Medicine for the United States and EDQM from the European Directorate for the Quality in Medicines and Healthcare for Europe.

BACKGROUND: There is currently no system that aligns pharmaceutically equivalent medicinal products across nations, creating obstacles to transnational medication prescribing and medical research. EDQM has been internationally recognized as the leading system in systematic pharmaceutical product descriptions. RxNorm is a critical terminology based in the US and used widely in applications internationally that would benefit from alignment with EDQM-based dosage form descriptions. GOAL: Demonstrate a method for alignment of RxNorm dosage forms with EDQM terminologies and with EDQM dosage forms. Describe obstacles and advantages of such an alignment for ultimate application in calculating universal Pharmaceutical Product Identifiers. METHODS: A pharmaceutical sciences student and a clinical pharmacology expert in dosage forms used definitions supplied by RxNorm and EDQM technical documentation to align the 120 RxNorm dose forms to EDQM-based dosage form description terms. The alignment of RxNorm to EDQM was then used to fit the RxNorm dose forms into an ontology based on EDQM. RESULTS AND CONCLUSIONS: The alignment of RxNorm and EDQM requires further validation but provides a potential method of establishing interoperability between the two terminologies without cumbersome manual reclassification. There remain ambiguities within each dosage form nomenclature that create obstacles to integrating medication databases rooted in EDQM and RxNorm into a single worldwide database.

Proposed Patient-Inclusive Methodology for Developing and Validating Patient Experience Surveys.

Patient experience while using telehealth services impacts patient care, and health system incentives and reimbursement. Therefore, the patient experience should be continuously improved, which requires evaluation. Surveying patients is integral to this process. Ideally, patient experience surveys used to evaluate telehealth services should cover patient experience sub-themes, be validated, and minimize bias. This paper evaluates how current validated patient experience surveys cover patient experience sub-themes, and how many are telehealth-specific. We collected 11 validated patient experience surveys. We then extracted five themes and 114 sub-themes of the patient experience. We evaluated survey questions against patient experience sub-themes. We found that current validated patient experience surveys cover, at most, 20.2% of patient experience sub-themes, with the most common sub-themes being "Psychosocial Needs" (81.8%) and "Information: Treatment" (72.7%). We cross-referenced validated patient experience surveys against validated telehealth-specific surveys. Only one validated patient experience survey (PEQ) was also telehealth-specific.

Review of the Performance Metrics for Natural Language Systems for Clinical Trials Matching.

Natural Language Processing (NLP) has been adopted widely in clinical trial matching for its ability to process unstructured text that is often found in electronic health records. Despite the rise in the new tools that use NLP to match patients to eligible clinical trials, the comparison of these tools is difficult due to the lack of consistency in how these tools are evaluated. The ground truth or reference that the tools use to assess results varies, making it difficult to compare the robustness of the tools against each other. This paper alarms the lack of definition and consistency of ground truth data used to evaluate such tools and suggests two ways to define a gold standard for the ground truth in small and large-scale studies.

Understanding the social determinants of health and genetic factors contributing to the differences observed in COVID-19 incidence and mortality between underrepresented and other communities.

COVID-19 has been a devastating disease, especially in underserved communities. Data has shown that Indigenous peoples, Latinx communities, and Black Americans have a 3.3, 2.4, and 2 times higher mortality rate than White communities, respectively, due to COVID-19. Therefore, in this paper, we sought to understand how Social Determinants of Health and genetic factors influence COVID-19 incidence, mortality rates, and complications by assessing existing literature. Studies showed that identifying with a racial/ethnic minority, being homeless, housing insecurity, lower household median income, and living in an area with decreased air quality were associated with higher incidence and mortality from COVID-19. Analyses of these studies also showed a lack of resources to collect patients' social determinants of health, revealing an urgent need to create databases with information on local support programs and operationalize the referral and tracking outcomes to address the health inequities for Black, Indigenous, and Latinx communities.