A community-of-practice-based evaluation methodology for knowledge intensive computational methods and its application to multimorbidity decision support.

OBJECTIVE: The study has dual objectives. Our first objective (1) is to develop a community-of-practice-based evaluation methodology for knowledge-intensive computational methods. We target a whitebox analysis of the computational methods to gain insight on their functional features and inner workings. In more detail, we aim to answer evaluation questions on (i) support offered by computational methods for functional features within the application domain; and (ii) in-depth characterizations of the underlying computational processes, models, data and knowledge of the computational methods. Our second objective (2) involves applying the evaluation methodology to answer questions (i) and (ii) for knowledge-intensive clinical decision support (CDS) methods, which operationalize clinical knowledge as computer interpretable guidelines (CIG); we focus on multimorbidity CIG-based clinical decision support (MGCDS) methods that target multimorbidity treatment plans. MATERIALS AND METHODS: Our methodology directly involves the research community of practice in (a) identifying functional features within the application domain; (b) defining exemplar case studies covering these features; and (c) solving the case studies using their developed computational methods-research groups detail their solutions and functional feature support in solution reports. Next, the study authors (d) perform a qualitative analysis of the solution reports, identifying and characterizing common themes (or dimensions) among the computational methods. This methodology is well suited to perform whitebox analysis, as it directly involves the respective developers in studying inner workings and feature support of computational methods. Moreover, the established evaluation parameters (e.g., features, case studies, themes) constitute a re-usable benchmark framework, which can be used to evaluate new computational methods as they are developed. We applied our community-of-practice-based evaluation methodology on MGCDS methods. RESULTS: Six research groups submitted comprehensive solution reports for the exemplar case studies. Solutions for two of these case studies were reported by all groups. We identified four evaluation dimensions: detection of adverse interactions, management strategy representation, implementation paradigms, and human-in-the-loop support. Based on our whitebox analysis, we present answers to the evaluation questions (i) and (ii) for MGCDS methods. DISCUSSION: The proposed evaluation methodology includes features of illuminative and comparison-based approaches; focusing on understanding rather than judging/scoring or identifying gaps in current methods. It involves answering evaluation questions with direct involvement of the research community of practice, who participate in setting up evaluation parameters and solving exemplar case studies. Our methodology was successfully applied to evaluate six MGCDS knowledge-intensive computational methods. We established that, while the evaluated methods provide a multifaceted set of solutions with different benefits and drawbacks, no single MGCDS method currently provides a comprehensive solution for MGCDS. CONCLUSION: We posit that our evaluation methodology, applied here to gain new insights into MGCDS, can be used to assess other types of knowledge-intensive computational methods and answer other types of evaluation questions. Our case studies can be accessed at our GitHub repository (https://github.com/william-vw/MGCDS).

Representation of evidence-based clinical practice guideline recommendations on FHIR.

BACKGROUND: Various formalisms have been developed to represent clinical practice guideline recommendations in a computer-interpretable way. However, none of the existing formalisms leverage the structured and computable information that emerge from the evidence-based guideline development process. Thus, we here propose a FHIR-based format that uses computer-interpretable representations of the knowledge artifacts that emerge during the process of evidence-based guideline development to directly serve as the basis of evidence-based recommendations. METHODS: We identified the information required to represent evidence-based clinical practice guideline recommendations and reviewed the knowledge artifacts emerging during the evidence-based guideline development process. We then conducted a consensus-based design process with domain experts to develop an information model for guideline recommendation representation that is structurally aligned to the evidence-based guideline recommendation development process and a corresponding representation based on FHIR resources developed for evidence-based medicine (EBMonFHIR). The resulting recommendations were modelled and represented in conformance with the FHIR Clinical Guidelines (CPG-on-FHIR) implementation guide. RESULTS: The information model of evidence-based clinical guideline recommendations and its EBMonFHIR-/CPG-on-FHIR-based representation contain the clinical contents of individual guideline recommendations, a set of metadata for the recommendations, the ratings for the recommendations (e.g., strength of recommendation, certainty of overall evidence), the ratings of certainty of evidence for individual outcomes (e.g., risk of bias) and links to the underlying evidence (systematic reviews based on primary studies). We created profiles and an implementation guide for all FHIR resources required to represent the knowledge artifacts generated during evidence-based guideline development and their re-use as the basis for recommendations and used the profiles to implement an exemplary clinical guideline recommendation. CONCLUSIONS: The FHIR implementation guide presented here can be used to directly link the evidence assessment process of evidence-based guideline recommendation development, i.e. systematic reviews and evidence grading, and the underlying evidence from primary studies to the resulting guideline recommendations. This not only allows the evidence on which recommendations are based on to be evaluated transparently and critically, but also enables guideline developers to leverage computable evidence in a more direct way to facilitate the generation of computer-interpretable guideline recommendations.

Conformance analysis for comorbid patients in Answer Set Programming.

The treatment of comorbid patients is a hot problem in Medical Informatics, since the plain application of multiple Computer-Interpretable Guidelines (CIGs) can lead to interactions that are potentially dangerous for the patients. The specialized literature has mostly focused on the "a priori" or "execution-time" analysis of the interactions between multiple Computer-Interpretable Guidelines (CIGs), and/or CIG "merge". In this paper, we face a complementary problem, namely, the a posteriori analysis of the treatment of comorbid patients. Given the CIGs, the history of the status of the patient, and the log of the clinical actions executed on her, we try to explain the actions executed on the patient (i.e., the log) in terms of the actions recommended by the CIGs, of their potential interactions, and of the possible ways of managing each such interaction, pointing out (i) deviations from CIG recommendations not explained in terms of interaction management (if any) and (ii) unmanaged interactions (if any). Our approach is based on Answer Set Programming, and, to face realistic problems, devotes specific attention to the temporal dimension.

Use of Real-World FHIR Data Combined with Context-Sensitive Decision Modeling to Guide Sentinel Biopsy in Melanoma.

Background: To support clinical decision-making at the point of care, the "best next step" based on Standard Operating Procedures (SOPs) and actual accurate patient data must be provided. To do this, textual SOPs have to be transformed into operable clinical algorithms and linked to the data of the patient being treated. For this linkage, we need to know exactly which data are needed by clinicians at a certain decision point and whether these data are available. These data might be identical to the data used within the SOP or might integrate a broader view. To address these concerns, we examined if the data used by the SOP is also complete from the point of view of physicians for contextual decision-making. Methods: We selected a cohort of 67 patients with stage III melanoma who had undergone adjuvant treatment and mainly had an indication for a sentinel biopsy. First, we performed a step-by-step simulation of the patient treatment along our clinical algorithm, which is based on a hospital-specific SOP, to validate the algorithm with the given Fast Healthcare Interoperability Resources (FHIR)-based data of our cohort. Second, we presented three different decision situations within our algorithm to 10 dermatooncologists, focusing on the concrete patient data used at this decision point. The results were conducted, analyzed, and compared with those of the pure algorithmic simulation. Results: The treatment paths of patients with melanoma could be retrospectively simulated along the clinical algorithm using data from the patients' electronic health records. The subsequent evaluation by dermatooncologists showed that the data used at the three decision points had a completeness between 84.6% and 100.0% compared with the data used by the SOP. At one decision point, data on "patient age (at primary diagnosis)" and "date of first diagnosis" were missing. Conclusions: The data needed for our decision points are available in the FHIR-based dataset. Furthermore, the data used at decision points by the SOP and hence the clinical algorithm are nearly complete compared with the data required by physicians in clinical practice. This is an important precondition for further research focusing on presenting decision points within a treatment process integrated with the patient data needed.

Computer-interpretable clinical guidelines: a methodological review.

Clinical practice guidelines (CPGs) aim to improve the quality of care, reduce unjustified practice variations and reduce healthcare costs. In order for them to be effective, clinical guidelines need to be integrated with the care flow and provide patient-specific advice when and where needed. Hence, their formalization as computer-interpretable guidelines (CIGs) makes it possible to develop CIG-based decision-support systems (DSSs), which have a better chance of impacting clinician behavior than narrative guidelines. This paper reviews the literature on CIG-related methodologies since the inception of CIGs, while focusing and drawing themes for classifying CIG research from CIG-related publications in the Journal of Biomedical Informatics (JBI). The themes span the entire life-cycle of CIG development and include: knowledge acquisition and specification for improved CIG design, including (1) CIG modeling languages and (2) CIG acquisition and specification methodologies, (3) integration of CIGs with electronic health records (EHRs) and organizational workflow, (4) CIG validation and verification, (5) CIG execution engines and supportive tools, (6) exception handling in CIGs, (7) CIG maintenance, including analyzing clinician's compliance to CIG recommendations and CIG versioning and evolution, and finally (8) CIG sharing. I examine the temporal trends in CIG-related research and discuss additional themes that were not identified in JBI papers, including existing themes such as overcoming implementation barriers, modeling clinical goals, and temporal expressions, as well as futuristic themes, such as patient-centric CIGs and distributed CIGs.

A process mining approach for clinical guidelines compliance: real-world application in rectal cancer.

In the era of evidence-based medicine, several clinical guidelines were developed, supporting cancer management from diagnosis to treatment and aiming to optimize patient care and hospital resources. Nevertheless, individual patient characteristics and organizational factors may lead to deviations from these standard recommendations during clinical practice. In this context, process mining in healthcare constitutes a valid tool to evaluate conformance of real treatment pathways, extracted from hospital data warehouses as event log, to standard clinical guidelines, translated into computer-interpretable formats. In this study we translate the European Society of Medical Oncology guidelines for rectal cancer treatment into a computer-interpretable format using Pseudo-Workflow formalism (PWF), a language already employed in pMineR software library for Process Mining in Healthcare. We investigate the adherence of a real-world cohort of rectal cancer patients treated at Fondazione Policlinico Universitario A. Gemelli IRCCS, data associated with cancer diagnosis and treatment are extracted from hospital databases in 453 patients diagnosed with rectal cancer. PWF enables the easy implementation of guidelines in a computer-interpretable format and visualizations that can improve understandability and interpretability of physicians. Results of the conformance checking analysis on our cohort identify a subgroup of patients receiving a long course treatment that deviates from guidelines due to a moderate increase in radiotherapy dose and an addition of oxaliplatin during chemotherapy treatment. This study demonstrates the importance of PWF to evaluate clinical guidelines adherence and to identify reasons of deviations during a treatment process in a real-world and multidisciplinary setting.

The use of computer-interpretable clinical guidelines to manage care complexities of patients with multimorbid conditions: A review.

Clinical practice guidelines (CPGs) document evidence-based information and recommendations on treatment and management of conditions. CPGs usually focus on management of a single condition; however, in many cases a patient will be at the centre of multiple health conditions (multimorbidity). Multiple CPGs need to be followed in parallel, each managing a separate condition, which often results in instructions that may interact with each other, such as conflicts in medication. Furthermore, the impetus to deliver customised care based on patient-specific information, results in the need to be able to offer guidelines in an integrated manner, identifying and managing their interactions. In recent years, CPGs have been formatted as computer-interpretable guidelines (CIGs). This enables developing CIG-driven clinical decision support systems (CDSSs), which allow the development of IT applications that contribute to the systematic and reliable management of multiple guidelines. This study focuses on understanding the use of CIG-based CDSSs, in order to manage care complexities of patients with multimorbidity. The literature between 2011 and 2017 is reviewed, which covers: (a) the challenges and barriers in the care of multimorbid patients, (b) the role of CIGs in CDSS augmented delivery of care, and (c) the approaches to alleviating care complexities of multimorbid patients. Generating integrated care plans, detecting and resolving adverse interactions between treatments and medications, dealing with temporal constraints in care steps, supporting patient-caregiver shared decision making and maintaining the continuity of care are some of the approaches that are enabled using a CIG-based CDSS.

Comparing a Mobile Decision Support System Versus the Use of Printed Materials for the Implementation of an Evidence-Based Recommendation: Protocol for a Qualitative Evaluation.

BACKGROUND: The distribution of printed materials is the most frequently used strategy to disseminate and implement clinical practice guidelines, although several studies have shown that the effectiveness of this approach is modest at best. Nevertheless, there is insufficient evidence to support the use of other strategies. Recent research has shown that the use of computerized decision support presents a promising approach to address some aspects of this problem. OBJECTIVE: The aim of this study is to provide qualitative evidence on the potential effect of mobile decision support systems to facilitate the implementation of evidence-based recommendations included in clinical practice guidelines. METHODS: We will conduct a qualitative study with two arms to compare the experience of primary care physicians while they try to implement an evidence-based recommendation in their clinical practice. In the first arm, we will provide participants with a printout of the guideline article containing the recommendation, while in the second arm, we will provide participants with a mobile app developed after formalizing the recommendation text into a clinical algorithm. Data will be collected using semistructured and open interviews to explore aspects of behavioral change and technology acceptance involved in the implementation process. The analysis will be comprised of two phases. During the first phase, we will conduct a template analysis to identify barriers and facilitators in each scenario. Then, during the second phase, we will contrast the findings from each arm to propose hypotheses about the potential impact of the system. RESULTS: We have formalized the narrative in the recommendation into a clinical algorithm and have developed a mobile app. Data collection is expected to occur during 2018, with the first phase of analysis running in parallel. The second phase is scheduled to conclude in July 2019. CONCLUSIONS: Our study will further the understanding of the role of mobile decision support systems in the implementation of clinical practice guidelines. Furthermore, we will provide qualitative evidence to aid decisions made by low- and middle-income countries' ministries of health about investments in these technologies.

Assessment of a personalized and distributed patient guidance system.

OBJECTIVES: The MobiGuide project aimed to establish a ubiquitous, user-friendly, patient-centered mobile decision-support system for patients and for their care providers, based on the continuous application of clinical guidelines and on semantically integrated electronic health records. Patients would be empowered by the system, which would enable them to lead their normal daily lives in their regular environment, while feeling safe, because their health state would be continuously monitored using mobile sensors and self-reporting of symptoms. When conditions occur that require medical attention, patients would be notified as to what they need to do, based on evidence-based guidelines, while their medical team would be informed appropriately, in parallel. We wanted to assess the system's feasibility and potential effects on patients and care providers in two different clinical domains. MATERIALS AND METHODS: We describe MobiGuide's architecture, which embodies these objectives. Our novel methodologies include a ubiquitous architecture, encompassing a knowledge elicitation process for parallel coordinated workflows for patients and care providers; the customization of computer-interpretable guidelines (CIGs) by secondary contexts affecting remote management and distributed decision-making; a mechanism for episodic, on demand projection of the relevant portions of CIGs from a centralized, backend decision-support system (DSS), to a local, mobile DSS, which continuously delivers the actual recommendations to the patient; shared decision-making that embodies patient preferences; semantic data integration; and patient and care provider notification services. MobiGuide has been implemented and assessed in a preliminary fashion in two domains: atrial fibrillation (AF), and gestational diabetes Mellitus (GDM). Ten AF patients used the AF MobiGuide system in Italy and 19 GDM patients used the GDM MobiGuide system in Spain. The evaluation of the MobiGuide system focused on patient and care providers' compliance to CIG recommendations and their satisfaction and quality of life. RESULTS: Our evaluation has demonstrated the system's capability for supporting distributed decision-making and its use by patients and clinicians. The results show that compliance of GDM patients to the most important monitoring targets - blood glucose levels (performance of four measurements a day: 0.87±0.11; measurement according to the recommended frequency of every day or twice a week: 0.99±0.03), ketonuria (0.98±0.03), and blood pressure (0.82±0.24) - was high in most GDM patients, while compliance of AF patients to the most important targets was quite high, considering the required ECG measurements (0.65±0.28) and blood-pressure measurements (0.75±1.33). This outcome was viewed by the clinicians as a major potential benefit of the system, and the patients have demonstrated that they are capable of self-monitoring - something that they had not experienced before. In addition, the system caused the clinicians managing the AF patients to change their diagnosis and subsequent treatment for two of the ten AF patients, and caused the clinicians managing the GDM patients to start insulin therapy earlier in two of the 19 patients, based on system's recommendations. Based on the end-of-study questionnaires, the sense of safety that the system has provided to the patients was its greatest asset. Analysis of the patients' quality of life (QoL) questionnaires for the AF patients was inconclusive, because while most patients reported an improvement in their quality of life in the EuroQoL questionnaire, most AF patients reported a deterioration in the AFEQT questionnaire. DISCUSSION: Feasibility and some of the potential benefits of an evidence-based distributed patient-guidance system were demonstrated in both clinical domains. The potential application of MobiGuide to other medical domains is supported by its standards-based patient health record with multiple electronic medical record linking capabilities, generic data insertion methods, generic medical knowledge representation and application methods, and the ability to communicate with a wide range of sensors. Future larger scale evaluations can assess the impact of such a system on clinical outcomes. CONCLUSION: MobiGuide's feasibility was demonstrated by a working prototype for the AF and GDM domains, which is usable by patients and clinicians, achieving high compliance to self-measurement recommendations, while enhancing the satisfaction of patients and care providers.

Temporal detection and analysis of guideline interactions.

BACKGROUND: Clinical practice guidelines (CPGs) are assuming a major role in the medical area, to grant the quality of medical assistance, supporting physicians with evidence-based information of interventions in the treatment of single pathologies. The treatment of patients affected by multiple diseases (comorbid patients) is one of the main challenges for the modern healthcare. It requires the development of new methodologies, supporting physicians in the treatment of interactions between CPGs. Several approaches have started to face such a challenging problem. However, they suffer from a substantial limitation: they do not take into account the temporal dimension. Indeed, practically speaking, interactions occur in time. For instance, the effects of two actions taken from different guidelines may potentially conflict, but practical conflicts happen only if the times of execution of such actions are such that their effects overlap in time. OBJECTIVES: We aim at devising a methodology to detect and analyse interactions between CPGs that considers the temporal dimension. METHODS: In this paper, we first extend our previous ontological model to deal with the fact that actions, goals, effects and interactions occur in time, and to model both qualitative and quantitative temporal constraints between them. Then, we identify different application scenarios, and, for each of them, we propose different types of facilities for user physicians, useful to support the temporal detection of interactions. RESULTS: We provide a modular approach in which different Artificial Intelligence temporal reasoning techniques, based on temporal constraint propagation, are widely exploited to provide users with such facilities. We applied our methodology to two cases of comorbidities, using simplified versions of CPGs. CONCLUSION: We propose an innovative approach to the detection and analysis of interactions between CPGs considering different sources of temporal information (CPGs, ontological knowledge and execution logs), which is the first one in the literature that takes into account the temporal issues, and accounts for different application scenarios.