Measured Performance and Healthcare Professional Perception of Large Language Models Used as Clinical Decision Support Systems: A Scoping Review.

The healthcare sector confronts challenges from overloaded tumor board meetings, reduced discussion durations, and care quality concerns, necessitating innovative solutions. Integrating Clinical Decision Support Systems (CDSSs) has a potential in supporting clinicians to reduce the cancer burden, but CDSSs remain poorly used in clinical practice. The emergence of OpenAI's ChatGPT in 2022 has prompted the evaluation of Large Language Models (LLMs) as potential CDSSs for diagnosis and therapeutic management. We conducted a scoping review to evaluate the utility of LLMs like ChatGPT as CDSSs in several medical specialties, particularly in oncology, and compared users' perception of LLMs with the actually measured performance of these systems.

Exploring Explainable AI Techniques for Text Classification in Healthcare: A Scoping Review.

Text classification plays an essential role in the medical domain by organizing and categorizing vast amounts of textual data through machine learning (ML) and deep learning (DL). The adoption of Artificial Intelligence (AI) technologies in healthcare has raised concerns about the interpretability of AI models, often perceived as "black boxes." Explainable AI (XAI) techniques aim to mitigate this issue by elucidating AI model decision-making process. In this paper, we present a scoping review exploring the application of different XAI techniques in medical text classification, identifying two main types: model-specific and model-agnostic methods. Despite some positive feedback from developers, formal evaluations with medical end users of these techniques remain limited. The review highlights the necessity for further research in XAI to enhance trust and transparency in AI-driven decision-making processes in healthcare.

Leveraging Rule-Based NLP to Translate Textual Reports as Structured Inputs Automatically Processed by a Clinical Decision Support System.

Using clinical decision support systems (CDSSs) for breast cancer management necessitates to extract relevant patient data from textual reports which is a complex task although efficiently achieved by machine learning but black box methods. We proposed a rule-based natural language processing (NLP) method to automate the translation of breast cancer patient summaries into structured patient profiles suitable for input into the guideline-based CDSS of the DESIREE project. Our method encompasses named entity recognition (NER), relation extraction and structured data extraction to systematically organize patient data. The method demonstrated strong alignment with treatment recommendations generated for manually created patient profiles (gold standard) with only 2% of differences. Moreover, the NER pipeline achieved an average F1-score of 0.9 across the main entities (patient, side, and tumor), of 0,87 for relation extraction, and 0.75 for contextual information, showing promising results for rule-based NLP.

Simultaneously authoring and modeling clinical practice guidelines: a case study in the therapeutic management of type 2 diabetes in France.

By providing patient-specific advice, clinical decision support systems (CDSSs) are expected to promote the implementation of clinical practice guidelines (CPGs) to improve the quality of care. However, produced as texts, often incomplete and ambiguous, CPGs are difficult to translate into the formal knowledge bases (KBs) of CDSSs. The French National Authority for Health (HAS) decided to update CPGs on the management of type 2 diabetes. This work illustrates the simultaneous development of the text and its formal counterpart in a CDSS named RecosDiab. CPGs were elaborated by a working group according to the guideline development methodology. Textual recommendations were graded, either as evidence-based when evidence existed or as consensus-based when acknowledge by the working group. Knowledge modeling was performed following the steps of de-abstraction, disambiguation, and verification of completeness. This last step generated clinical situations not explicitly mentioned in the text and were graded as expert-based. The resulting KB provides therapeutic advice for 805 clinical situations, among which 2 are graded as evidence-based, 37 are consensus-based, and 766 are expert-based. However, because of the amount of expert-based propositions, the HAS did not endorse the system.

Clinical Decision Support Systems Applied to the Management of Breast Cancer Patients: A Scoping Review.

Breast cancer is the most commonly diagnosed cancer worldwide, and its burden has been rising over the past decades. A significant advance in healthcare is the integration of Clinical Decision Support Systems (CDSSs) into medical practice, which support healthcare professionals improving clinical decisions, leading to recommended patient-specific treatments and enhanced patient care. Breast cancer CDSSs are thus currently expanding, whether applied to screening, diagnostic, therapeutic or follow-up tasks. We conducted a scoping review to study their availability and use in practice. Except risk calculators, very few CDSSs are currently routinely used.

Elicitating patient patterns of physician non-compliance with breast cancer guidelines using formal concept analysis.

Because they provide patient-specific guideline-based recommendations, clinical decision support systems (CDSSs) are expected to promote the implementation of clinical practice guidelines (CPGs). OncoDoc2 is a CDSS applied to the management of breast cancer. However, despite it was routinely used during weekly multidisciplinary staff meetings (MSMs) at the Tenon Hospital (Paris, France), the compliance rate of MSMs' decisions with CPGs did not reach 100%. Formal Concept Analysis (FCA) has been applied to elicit formal concepts related to non-compliance. A statistical pre-treatment of attributes has been proposed to leverage FCA and discriminate between compliant and non-compliant decisions. Among the 1,889 decisions made over a 3 year-period, 199 decisions of recommended re-excisions have been considered for analysis. In this sample, non-compliance was explained by uncommon clinical profiles and specific patient-centred clinical criteria.

Accuracy of clinical data entry when using a computerized decision support system: a case study with OncoDoc2.

Some studies suggest that the implementation of health information technology (HIT) introduces unpredicted and unintended consequences including e-iatrogenesis. OncoDoc2 is a guideline-based clinical decision support system (CDSS) applied to the management of breast cancer. The system is used by answering closed-ended questions in order to document patient data while navigating through the knowledge base until the best patient-specific recommended treatments are obtained. OncoDoc2 has been used by three hospitals in real clinical settings and for genuine patients. We analysed 394 navigations, recorded on a 10-month period, which correspond to 6,025 data entries. The data entry error rate is 4.2%, spread over 52% of incorrect navigations (N-). However, the overall compliance rate of clinical decisions with guidelines significantly increased from 72.8% (without CDSS) to 87.3% (with CDSS). Although this increase is lowered because of N- navigations (compliance rates are respectively 95% and 80% for N+ and N- navigations), the benefits of HIT outweighted its disadvantages in our study.

Using Machine Learning and Deep Learning Methods to Predict the Complexity of Breast Cancer Cases.

In many countries, the management of cancer patients must be discussed in multidisciplinary tumor boards (MTBs). These meetings have been introduced to provide a collaborative and multidisciplinary approach to cancer care. However, the benefits of MTBs are now being challenged because there are a lot of cases and not enough time to discuss all the of them. During the evaluation of the guideline-based clinical decision support system (CDSS) of the DESIREE project, we found that for some clinical cases, the system did not produce recommendations. We assumed that these cases were complex clinical cases and needed deeper MTB discussions. In this work, we trained and tested several machine learning and deep learning algorithms on a labelled sample of 298 breast cancer patient summaries, to predict the complexity of a breast cancer clinical case. XGboost and multi-layer perceptron were the models with the best result, with an F1 score of 83%.

Comparison of MetaMap, cTAKES, SIFR, and ECMT to Annotate Breast Cancer Patient Summaries.

Most clinical texts including breast cancer patient summaries (BCPSs) are elaborated as narrative documents difficult to process by decision support systems. Annotators have been developed to extract the relevant content of such documents, e.g., MetaMap and cTAKES, that work with the English language and perform concept mapping using UMLS, SIFR and ECMT, that work for the French language and provide concepts using various terminologies. We compared the four annotators on a sample of 25 French BCPSs, pre-processed to manage acronyms and translated in English. We observed that MetaMap extracted the largest number of UMLS concepts (15,458), followed by SIFR (3,784), ECMT (1,962), and cTAKES (1,769). Each annotator extracted specific valuable information, not proposed by the other annotators. Considered as complementary, all annotators should be used in sequence to optimize the results.

Methods Used to Compare Narrative Clinical Practice Guidelines: A Scoping Review.

Guideline-based clinical decision support systems (CDSSs) need the most recent evidence for reliable performance, making the provision of regularly updated clinical practice guidelines (CPGs) a major issue. Some international guidelines are renewed in short intervals and can be used for checking the status of given national guidelines with regard to the most recent evidence. Considering the volume of medical data and the number of CPGs published, computerized comparison of clinical guidelines can be an effective method. We performed a scoping review to evaluate the methods used for comparing two CPGs. We searched for methods for extracting CPG components and for methods used for comparing CPGs at different levels of abstraction. In each case, computerized and semi-computerized methods were recognized. Expert knowledge has yet a determinant role for assessing the comparisons, this role being more prominent for the extraction of semantic rules and the resolution of inconsistencies.

Using Machine Learning on Imbalanced Guideline Compliance Data to Optimize Multidisciplinary Tumour Board Decision Making for the Management of Breast Cancer Patients.

Complex breast cancer cases that need further multidisciplinary tumor board (MTB) discussions should have priority in the organization of MTBs. In order to optimize MTB workflow, we attempted to predict complex cases defined as non-compliant cases despite the use of the decision support system OncoDoc, through the implementation of machine learning procedures and algorithms (Decision Trees, Random Forests, and XGBoost). F1-score after cross-validation, sampling implementation, with or without feature selection, did not exceed 40%.

ABiMed: Towards an Innovative Clinical Decision Support System for Medication Reviews and Polypharmacy Management.

Polypharmacy in elderly is a public health problem with both clinical (increase of adverse drug events) and economic issues. One solution is medication review, a structured assessment of patients' drug orders by the pharmacist for optimizing the therapy. However, this task is tedious, cognitively complex and error-prone, and only a few clinical decision support systems have been proposed for supporting it. Existing systems are either rule-based systems implementing guidelines, or documentary systems presenting drug knowledge. In this paper, we present the ABiMed research project, and, through literature reviews and brainstorming, we identified five candidate innovations for a decision support system for medication review: patient data transfer from GP to pharmacists, use of semantic technologies, association of rule-based and documentary approaches, use of machine learning, and a two-way discussion between pharmacist and GP after the medication review.

Computing the compliance of physician drug orders with guidelines using an OWL2 reasoner and standard drug resources.

Assessing the conformity of a physician's prescription to a given recommended prescription is not obvious since both prescriptions are expressed at different levels of abstraction and may concern only a subpart of the whole order. Recent formalisms (OWL2) and tools (reasoners) from the semantic web technologies are becoming available to represent defined concepts and to handle classification services. We propose a generic framework based on such technologies, using available standardized drug resources, to compute the compliance of a given drug order to a recommended prescription, such that the subsumption relationship yields the conformity relationship between the order and the recommendation. The ATC drug classification has been used as a local ontology. The method has been successfully implemented for arterial hypertension management for which we had a sample of antihypertensive orders. However, supplemental standardized drug knowledge is needed to correctly compare drug orders to recommended orders.

A generic system for critiquing physicians' prescriptions: usability, satisfaction and lessons learnt.

Clinical decision support systems have been developed to help physicians to take clinical guidelines into account during consultations. The ASTI critiquing module is one such systems; it provides the physician with automatic criticisms when a drug prescription does not follow the guidelines. It was initially developed for hypertension and type 2 diabetes, but is designed to be generic enough for application to all chronic diseases. We present here the results of usability and satisfaction evaluations for the ASTI critiquing module, obtained with GPs for a newly implemented guideline concerning dyslipaemia, and we discuss the lessons learnt and the difficulties encountered when building a generic DSS for critiquing physicians' prescriptions.

Are Semantic Annotators Able to Extract Relevant Complexity-Related Concepts from Clinical Notes?

Clinical decision support systems (CDSSs) implementing cancer clinical practice guidelines (CPGs) have the potential to improve the compliance of decisions made by multidisciplinary tumor boards (MTB) with CPGs. However, guideline-based CDSSs do not cover complex cases and need time for discussion. We propose to learn how to predict complex cancer cases prior to MTBs from breast cancer patient summaries (BCPSs) resuming clinical notes. BCPSs being unstructured natural language textual documents, we implemented four semantic annotators (ECMT, SIFR, cTAKES, and MetaMap) to assess whether complexity-related concepts could be extracted from clinical notes. On a sample of 24 BCPSs covering 35 complexity reasons, ECMT and MetaMap were the most efficient systems with a performance rate of 60% (21/35) and 49% (17/35), respectively. When using the four annotators in sequence, 69% of complexity reasons were extracted (24/35 reasons).

Deep Learning, a Not so Magical Problem Solver: A Case Study with Predicting the Complexity of Breast Cancer Cases.

Using guideline-based clinical decision support systems (CDSSs) has improved clinical practice, especially during multidisciplinary tumour boards (MTBs) in cancer patient management. However, MTBs have been reported to be overcrowded, with limited time to discuss all cases. Complex breast cancer cases that need further MTB discussions should have priority in the organization of MTBs. In order to optimize MTB workflow, we attempted to predict complex cases defined as non-compliant cases despite the use of the decision support system OncoDoc. After previously obtaining insufficient performance with machine learning algorithms, we tested Multi Layer Perceptron for classification, compared various samplers to compensate data imbalance combined with cross- validation, and optimized all models with hyperparameter tuning and feature selection with no improvement and lacklustre results (F1-score: 31.4%).

Reusing Decisions Made with One Decision Support System to Assess a Second Decision Support System: Introducing the Notion of Complex Cases.

The guideline-based decision support system (GL-DSS) of the DESIREE project and OncoDoc are two clinical decision support systems applied to the management of breast cancer. In order to evaluate the DESIREE GL-DSS, we decided to reuse a sample of clinical cases previously resolved by the multidisciplinary tumor board (MTB) of the Tenon Hospital (Paris, France) when using OncoDoc. Since we had two different knowledge representation models to represent clinical parameters and decisions, and two formalisms to represent guidelines, we developed a transformation sequence, involving the creation of synthetic patients, the enrichment of DESIREE ontology, and the translation of clinical cases and their decisions, to transform OncoDoc data into the DESIREE representation. Considering MTB decisions as the gold standard, the 84% compliance rate of DESIREE recommendations was rather satisfactory. Some situations (0.7%) concerned clinical cases that were compliant neither with OncoDoc nor with DESIREE that we defined as complex cases, not handled by guidelines, which necessitate effective MTB discussions.

Users' Assessement of the Visualization with Gauges of a Guideline-Based Decision Support System Output: A Qualitative Study of Focus Groups.

Computerized decision support systems (CDSSs) are still poorly routinely implemented in clinical practices mainly because of usability problems related to the technology interface. We previously proposed to use gauges to visualize the output of a guideline-based CDSS applied to malnutrition and pressure ulcer management in nursing homes. This interface was assessed by four focus groups including 16 healthcare professionals with expertise in geriatrics. A USE-like questionnaire was distributed. Participants considered the dashboard-with-gauges visualization was useful (94%), easy to use (63%), easy to learn (88%), and 88% thought they could be satisfied with it. However, concerns were expressed about the difficulty to follow up multiple healthcare problems.

Computerized Decision Support Systems for Nursing Homes: A Scoping Review.

OBJECTIVES: To summarize the research literature describing the outcomes of computerized decision support systems (CDSSs) implemented in nursing homes (NHs). DESIGN: Scoping review. METHODS: Search of relevant articles published in the English language between January 1, 2000, and February 29, 2020, in the Medline database. The quality of the selected studies was assessed according to PRISMA guidelines and the Mixed Method Appraisal Tool. RESULTS: From 1828 articles retrieved, 24 studies were selected for review, among which only 6 were randomized controlled trials. Although clinical outcomes are seldom studied, some studies show that CDSSs have the potential to decrease pressure ulcer incidence and malnutrition prevalence. Improvement of process outcomes such as increased compliance with practice guidelines, better documentation of nursing assessment, improved teamwork and communication, and cost saving, also are reported. CONCLUSIONS AND IMPLICATIONS: Overall, the use of CDSSs in NHs may be effective to improve patient clinical outcomes and health care delivery; however, most of the retrieved studies were observational studies, which significantly weakens the evidence. High-quality studies are needed to investigate CDSS effects and limitations in NHs.

How to translate therapeutic recommendations in clinical practice guidelines into rules for critiquing physician prescriptions? Methods and application to five guidelines.

BACKGROUND: Clinical practice guidelines give recommendations about what to do in various medical situations, including therapeutical recommendations for drug prescription. An effective way to computerize these recommendations is to design critiquing decision support systems, i.e. systems that criticize the physician's prescription when it does not conform to the guidelines. These systems are commonly based on a list of "if conditions then criticism" rules. However, writing these rules from the guidelines is not a trivial task. The objective of this article is to propose methods that (1) simplify the implementation of guidelines' therapeutical recommendations in critiquing systems by automatically translating structured therapeutical recommendations into a list of "if conditions then criticize" rules, and (2) can generate an appropriate textual label to explain to the physician why his/her prescription is not recommended. METHODS: We worked on the therapeutic recommendations in five clinical practice guidelines concerning chronic diseases related to the management of cardiovascular risk. We evaluated the system using a test base of more than 2000 cases. RESULTS: Algorithms for automatically translating therapeutical recommendations into "if conditions then criticize" rules are presented. Eight generic recommendations are also proposed; they are guideline-independent, and can be used as default behaviour for handling various situations that are usually implicit in the guidelines, such as decreasing the dose of a poorly tolerated drug. Finally, we provide models and methods for generating a human-readable textual critique. The system was successfully evaluated on the test base. CONCLUSION: We show that it is possible to criticize physicians' prescriptions starting from a structured clinical guideline, and to provide clear explanations. We are now planning a randomized clinical trial to evaluate the impact of the system on practices.