A community-of-practice-based evaluation methodology for knowledge intensive computational methods and its application to multimorbidity decision support.

OBJECTIVE: The study has dual objectives. Our first objective (1) is to develop a community-of-practice-based evaluation methodology for knowledge-intensive computational methods. We target a whitebox analysis of the computational methods to gain insight on their functional features and inner workings. In more detail, we aim to answer evaluation questions on (i) support offered by computational methods for functional features within the application domain; and (ii) in-depth characterizations of the underlying computational processes, models, data and knowledge of the computational methods. Our second objective (2) involves applying the evaluation methodology to answer questions (i) and (ii) for knowledge-intensive clinical decision support (CDS) methods, which operationalize clinical knowledge as computer interpretable guidelines (CIG); we focus on multimorbidity CIG-based clinical decision support (MGCDS) methods that target multimorbidity treatment plans. MATERIALS AND METHODS: Our methodology directly involves the research community of practice in (a) identifying functional features within the application domain; (b) defining exemplar case studies covering these features; and (c) solving the case studies using their developed computational methods-research groups detail their solutions and functional feature support in solution reports. Next, the study authors (d) perform a qualitative analysis of the solution reports, identifying and characterizing common themes (or dimensions) among the computational methods. This methodology is well suited to perform whitebox analysis, as it directly involves the respective developers in studying inner workings and feature support of computational methods. Moreover, the established evaluation parameters (e.g., features, case studies, themes) constitute a re-usable benchmark framework, which can be used to evaluate new computational methods as they are developed. We applied our community-of-practice-based evaluation methodology on MGCDS methods. RESULTS: Six research groups submitted comprehensive solution reports for the exemplar case studies. Solutions for two of these case studies were reported by all groups. We identified four evaluation dimensions: detection of adverse interactions, management strategy representation, implementation paradigms, and human-in-the-loop support. Based on our whitebox analysis, we present answers to the evaluation questions (i) and (ii) for MGCDS methods. DISCUSSION: The proposed evaluation methodology includes features of illuminative and comparison-based approaches; focusing on understanding rather than judging/scoring or identifying gaps in current methods. It involves answering evaluation questions with direct involvement of the research community of practice, who participate in setting up evaluation parameters and solving exemplar case studies. Our methodology was successfully applied to evaluate six MGCDS knowledge-intensive computational methods. We established that, while the evaluated methods provide a multifaceted set of solutions with different benefits and drawbacks, no single MGCDS method currently provides a comprehensive solution for MGCDS. CONCLUSION: We posit that our evaluation methodology, applied here to gain new insights into MGCDS, can be used to assess other types of knowledge-intensive computational methods and answer other types of evaluation questions. Our case studies can be accessed at our GitHub repository (https://github.com/william-vw/MGCDS).

Supporting physicians in the coordination of distributed execution of CIGs to treat comorbid patients.

Clinical Practice Guidelines (CPGs) encode the "best" medical practices to treat patients affected by a specific disease and are widely used in the medical practice. Starting from the '90s', several Computer-Interpretable Guideline (CIG) systems have been devised to provide physicians with CPG-based decision support. CPGs (and CIGs) are devoted to provide evidence-based recommendations for one specific disease. In order to support the treatment of patients affected by multiple diseases (i.e., comorbid patients), challenging additional tasks have to be addressed, such as (i) the detection of the interactions between CIG actions, (ii) their management, and, finally, (iii) the "merge" or conciliation of the CIGs. Several CIG approaches have been recently extended in order to face (at least one of) such challenging problems, and one of them is GLARE. However, besides the solutions to tasks (i)-(iii) above, the "run-time" support to physicians treating a comorbid patient requires additional capabilities, to support the distribution of the management of interactions and of the execution of CIGs among different physicians. In this paper, we propose a general framework, based on GLARE and GLARE-SSCPM, to provide such additional capabilities.

Conformance analysis for comorbid patients in Answer Set Programming.

The treatment of comorbid patients is a hot problem in Medical Informatics, since the plain application of multiple Computer-Interpretable Guidelines (CIGs) can lead to interactions that are potentially dangerous for the patients. The specialized literature has mostly focused on the "a priori" or "execution-time" analysis of the interactions between multiple Computer-Interpretable Guidelines (CIGs), and/or CIG "merge". In this paper, we face a complementary problem, namely, the a posteriori analysis of the treatment of comorbid patients. Given the CIGs, the history of the status of the patient, and the log of the clinical actions executed on her, we try to explain the actions executed on the patient (i.e., the log) in terms of the actions recommended by the CIGs, of their potential interactions, and of the possible ways of managing each such interaction, pointing out (i) deviations from CIG recommendations not explained in terms of interaction management (if any) and (ii) unmanaged interactions (if any). Our approach is based on Answer Set Programming, and, to face realistic problems, devotes specific attention to the temporal dimension.

Supporting the distributed execution of clinical guidelines by multiple agents.

Clinical guidelines (GLs) are widely adopted in order to improve the quality of patient care, and to optimize it. To achieve such goals, their application on a specific patient usually requires the interventions of different agents, with different roles (e.g., physician, nurse), abilities (e.g., specialist in the treatment of alcohol-related problems) and contexts (e.g., many chronic patients may be treated at home). Additionally, the responsibility of the application of a guideline to a patient is usually retained by a physician, but delegation of responsibility (of the whole guideline, or of a part of it) is often used\required (e.g., delegation to a specialist), as well as the possibility, for a responsible, to select the executor of an action (e.g., a physician may retain the responsibility of an action, but delegate to a nurse its execution). To manage such phenomena, proper support to agent interaction and communication must be provided, providing agents with facilities for (1) treatment continuity (2) contextualization, (3) responsibility assignment and delegation (4) check of agent "appropriateness". In this paper we extend GLARE, a computerized GL management system, to support such needs. We illustrate our approach by means of a practical case study.

Representing and querying now-relative relational medical data.

Temporal information plays a crucial role in medicine. Patients' clinical records are intrinsically temporal. Thus, in Medical Informatics there is an increasing need to store, support and query temporal data (particularly in relational databases), in order, for instance, to supplement decision-support systems. In this paper, we show that current approaches to relational data have remarkable limitations in the treatment of "now-relative" data (i.e., data holding true at the current time). This can severely compromise their applicability in general, and specifically in the medical context, where "now-relative" data are essential to assess the current status of the patients. We propose a theoretically grounded and application-independent relational approach to cope with now-relative data (which can be paired, e.g., with different decision support systems) overcoming such limitations. We propose a new temporal relational representation, which is the first relational model coping with the temporal indeterminacy intrinsic in now-relative data. We also propose new temporal algebraic operators to query them, supporting the distinction between possible and necessary time, and Allen's temporal relations between data. We exemplify the impact of our approach, and study the theoretical and computational properties of the new representation and algebra.

Temporal detection and analysis of guideline interactions.

BACKGROUND: Clinical practice guidelines (CPGs) are assuming a major role in the medical area, to grant the quality of medical assistance, supporting physicians with evidence-based information of interventions in the treatment of single pathologies. The treatment of patients affected by multiple diseases (comorbid patients) is one of the main challenges for the modern healthcare. It requires the development of new methodologies, supporting physicians in the treatment of interactions between CPGs. Several approaches have started to face such a challenging problem. However, they suffer from a substantial limitation: they do not take into account the temporal dimension. Indeed, practically speaking, interactions occur in time. For instance, the effects of two actions taken from different guidelines may potentially conflict, but practical conflicts happen only if the times of execution of such actions are such that their effects overlap in time. OBJECTIVES: We aim at devising a methodology to detect and analyse interactions between CPGs that considers the temporal dimension. METHODS: In this paper, we first extend our previous ontological model to deal with the fact that actions, goals, effects and interactions occur in time, and to model both qualitative and quantitative temporal constraints between them. Then, we identify different application scenarios, and, for each of them, we propose different types of facilities for user physicians, useful to support the temporal detection of interactions. RESULTS: We provide a modular approach in which different Artificial Intelligence temporal reasoning techniques, based on temporal constraint propagation, are widely exploited to provide users with such facilities. We applied our methodology to two cases of comorbidities, using simplified versions of CPGs. CONCLUSION: We propose an innovative approach to the detection and analysis of interactions between CPGs considering different sources of temporal information (CPGs, ontological knowledge and execution logs), which is the first one in the literature that takes into account the temporal issues, and accounts for different application scenarios.

META-GLARE: A meta-system for defining your own computer interpretable guideline system-Architecture and acquisition.

CONTEXT: Several different computer-assisted management systems of computer interpretable guidelines (CIGs) have been developed by the Artificial Intelligence in Medicine community. Each CIG system is characterized by a specific formalism to represent CIGs, and usually provides a manager to acquire, consult and execute them. Though there are several commonalities between most formalisms in the literature, each formalism has its own peculiarities. OBJECTIVE: The goal of our work is to provide a flexible support to the extension or definition of CIGs formalisms, and of their acquisition and execution engines. Instead of defining "yet another CIG formalism and its manager", we propose META-GLARE (META Guideline Acquisition, Representation, and Execution), a "meta"-system to define new CIG systems. METHOD AND MATERIALS: In this paper, META-GLARE, a meta-system to define new CIG systems, is presented. We try to capture the commonalities among current CIG approaches, by providing (i) a general manager for the acquisition, consultation and execution of hierarchical graphs (representing the control flow of actions in CIGs), parameterized over the types of nodes and of arcs constituting it, and (ii) a library of different elementary components of guidelines nodes (actions) and arcs, in which each type definition involves the specification of how objects of this type can be acquired, consulted and executed. We provide generality and flexibility, by allowing free aggregations of such elementary components to define new primitive node and arc types. RESULTS: We have drawn several experiments, in which we have used META-GLARE to build a CIG system (Experiment 1 in Section 8), or to extend it (Experiments 2 and 3). Such experiments show that META-GLARE provides a useful and easy-to-use support to such tasks. For instance, re-building the Guideline Acquisition, Representation, and Execution (GLARE) system using META-GLARE required less than one day (Experiment 1). CONCLUSIONS: META-GLARE is a meta-system for CIGs supporting fast prototyping. Since META-GLARE provides acquisition and execution engines that are parametric over the specific CIG formalism, it supports easy update and construction of CIG systems.

META-GLARE: a shell for CIG systems.

In the last twenty years, many different approaches to deal with Computer-Interpretable clinical Guidelines (CIGs) have been developed, each one proposing its own representation formalism (mostly based on the Task-Network Model) execution engine. We propose META-GLARE a shell for easily defining new CIG systems. Using META-GLARE, CIG system designers can easily define their own systems (basically by defining their representation language), with a minimal programming effort. META-GLARE is thus a flexible and powerful vehicle for research about CIGs, since it supports easy and fast prototyping of new CIG systems.

Supporting a distributed execution of clinical guidelines.

Clinical guidelines (GL) play an important role in medical practice: the one of optimizing the quality of patient care on the basis of evidence based medicine. In order to achieve this goal, the interaction between different agents, who cooperate in the execution of the same GL, is a crucial issue. As a matter of fact, in many cases (e.g. in chronic disorders) the GL execution requires that patient treatment is not performed/completed in the hospital, but is continued in different contexts (e.g. at home, or in the general practitioner's ambulatory), under the responsibility of different agents. In this situation, the correct interaction and communication between the agents themselves is critical for the quality of care, and human resources coordination is a key issue to be addressed by the managers of the involved healthcare services. In this paper we describe how GLARE (Guideline Acquisition, Representation, and Execution), a computerized GL management system, has been extended in order to support such a need. In particular, we have provided: (i) an extension to GL actions representation languages, (ii) proper scheduling and (iii) querying services. By means of these enhancements we aimed at guaranteeing (1) treatment continuity and (2) responsibility assignment support in the various steps of a coordinated and distributed patient care process. We illustrate our approach by means of a practical case study.

Managing proposals and evaluations of updates to medical knowledge: theory and applications.

The process of keeping up-to-date the medical knowledge stored in relational databases is of paramount importance. Since quality and reliability of medical knowledge are essential, in many cases physicians' proposals of updates must undergo experts' evaluation before possibly becoming effective. However, until now no theoretical framework has been provided in order to cope with this phenomenon in a principled and non-ad hoc way. Indeed, such a framework is important not only in the medical domain, but in all Wikipedia-like contexts in which evaluation of update proposals is required. In this paper we propose GPVM (General Proposal Vetting Model), a general model to cope with update proposal⧹evaluation in relational databases. GPVM extends the current theory of temporal relational databases and, in particular, BCDM - Bitemporal Conceptual Data Model - "consensus" model, providing a new data model, new operations to propose and accept⧹reject updates, and new algebraic operators to query proposals. The properties of GPVM are also studied. In particular, GPVM is a consistent extension of BCDM and it is reducible to it. These properties ensure consistency with most relational temporal database frameworks, facilitating implementation on top of current frameworks and interoperability with previous approaches.

An implicit approach to deal with periodically repeated medical data.

CONTEXT: Temporal information plays a crucial role in medicine, so that in medical informatics there is an increasing awareness that suitable database approaches are needed to store and support it. Specifically, a great amount of clinical data (e.g., therapeutic data) are periodically repeated. Although an explicit treatment is possible in most cases, it causes severe storage and disk I/O problems. OBJECTIVE: In this paper, we propose an innovative approach to cope with periodic relational medical data in an implicit way. METHODS: We propose a new data model, representing periodic data in a compact (implicit) way, which is a consistent extension of TSQL2 consensus approach. Then, we identify some important types of temporal queries, and present query answering algorithms to answer them. Finally, we also run experiments to evaluate our approach. RESULTS: The experiments show that our approach outperforms current explicit approaches, especially as regard disk I/O. CONCLUSION: We have provided an implicit approach to periodic data with is a consistent extension of TSQL2 (and which is thus grant interoperable with it), and we have experimentally proven that it outperforms current explicit approaches.

Exceptions handling within GLARE clinical guideline framework.

Computerized clinical guidelines (CIGs) are widely adopted in order to assist practitioner and patient decision making. However, a main problem in their adoption is the fact that, during guidelines executions on specific patients, unpredictable facts and conditions (henceforth called exceptions) may occur. A proper and immediate treatment of such exception is necessary, but most current software systems coping with CIGs do not support it. In this paper, we describe how the GLARE system has been extended to deal with exceptions in CIGs.

Supporting human interaction and human resources coordination in distributed clinical guidelines.

Clinical guidelines (GL) play an important role in medical practice: the one of optimizing the quality of patient care on the basis of the best and most recent evidence based medicine. In order to achieve this goal, the interaction between different actors, who cooperate in the execution of the same GL, is a crucial issue. As a matter of fact, in many cases (e.g. in chronic disease treatment) the GL execution requires that patient treatment is not performed/completed in the hospital, but is continued in different contexts (e.g. at home, or in the general practitioner's ambulatory), under the responsibility of different actors. In this situation, the correct interaction and communication between the actors themselves is critical for the quality of care, and human resources coordination is a key issue to be addressed by the managers of the involved healthcare service. In this paper we describe how computerized GL management can be extended in order to support such needs, and we illustrate our approach by means of a practical case study.

Towards an implicit treatment of periodically-repeated medical data.

Temporal information plays a crucial role in medicine, so that in Medical Informatics there is an increasing awareness that suitable database approaches are needed to store and support it. Specifically, a great amount of clinical data (e.g., therapeutic data) are periodically repeated. Although an explicit treatment is possible in most cases, it causes severe storage and disk I/O problems. In this paper, we propose an innovative approach to cope with periodic medical data in an implicit way. We propose a new data model, representing periodic data in a compact (implicit) way, which is a consistent extension of TSQL2 consensus approach. Then, we identify some important types of temporal queries, and present query answering algorithms to answer them. We also sketch a temporal relational algebra for our approach. Finally, we show experimentally that our approach outperforms current explicit approaches.

Adopting model checking techniques for clinical guidelines verification.

OBJECTIVES: Clinical guidelines (GLs) are assuming a major role in the medical area, in order to grant the quality of the medical assistance and to optimize medical treatments within healthcare organizations. The verification of properties of the GL (e.g., the verification of GL correctness with respect to several criteria) is a demanding task, which may be enhanced through the adoption of advanced Artificial Intelligence techniques. In this paper, we propose a general and flexible approach to address such a task. METHODS AND MATERIALS: Our approach to GL verification is based on the integration of a computerized GL management system with a model-checker. We propose a general methodology, and we instantiate it by loosely coupling GLARE, our system for acquiring, representing and executing GLs, with the model-checker SPIN. RESULTS: We have carried out an in-depth analysis of the types of properties that can be effectively verified using our approach, and we have completed an overview of the usefulness of the verification task at the different stages of the GL life-cycle. In particular, experimentation on a GL for ischemic stroke has shown that the automatic verification of properties in the model checking approach is able to discover inconsistencies in the GL that cannot be detected in advance by hand. CONCLUSION: Our approach thus represents a further step in the direction of general and flexible automated GL verification, which also meets usability requirements.

An italian multicenter study for application of a diagnostic algorithm in autoantibody testing.

The presence in the serum of specific autoantibodies, such as antinuclear antibodies (ANA), anti-double-stranded DNA (anti-dsDNA), and antiextractable nuclear antigens (anti-ENA), is one of the diagnostic criteria for autoimmune rheumatic disease, and the requests for these tests in the last few years have grown remarkably. A guideline for reducing clinically inappropriate requests in autoantibody testing (ANA, anti-dsDNA, anti-ENA) has been applied in the Parma Hospital since 2007. The results for the period January-December 2007 were compared to those of the previous period January-December 2006, and a significant reduction in the number of anti-dsDNA (23.9%) and anti-ENA (20.7%) was found. The aim of this study was to assess the applicability of a similar guideline in a wide area (Parma, Modena, Piacenza, Reggio-Emilia) with reference to the diagnosis of autoimmune rheumatic disease. This project, supported by a regional grant for innovative research projects, was started in January 2008 and consists of three different steps: (1) a study group of clinicians and laboratory physicians to evaluate the diagnostic criteria, the analytical procedures, and the number of tests performed in different hospitals; (2) developing common guidelines for autoantibody testing that takes into account the different clinical needs with the aim of improving efficiency and clinical effectiveness of diagnosis and monitoring; and (3) assessing compliance with the guidelines in the different hospitals that are evaluating the second-level test (anti-dsDNA, anti-ENA) decrease. We think that the validation of guidelines for the laboratory diagnosis of autoimmune rheumatic disease can represent a tool for improving patients' outcomes and economic efficiency.

Coping efficiently with now-relative medical data.

In Medical Informatics, there is an increasing awareness that temporal information plays a crucial role, so that suitable database approaches are needed to store and support it. Specifically, most clinical data are intrinsically temporal, and a relevant part of them are now-relative (i.e., they are valid at the current time). Even if previous studies indicate that the treatment of now-relative data has a crucial impact on efficiency, current approaches have several limitations. In this paper we propose a novel approach, which is based on a new representation of now, and on query transformations. We also experimentally demonstrate that our approach outperforms its best competitors in the literature to the extent of a factor of more than ten, both in number of disk accesses and of CPU usage.

The temporal aspects of clinical guidelines.

Temporal aspects play a major role within clinical guidelines. Temporal issues arise when considering both guidelines per se, and the application of guidelines to specific patients. As a matter of fact, guidelines per se specify different diagnostic and\or therapeutic patterns, and temporal constraints on the intended times of execution of the actions they contain are an intrinsic part of guidelines themselves. Moreover, guidelines must be executed on the basis of patients' data, which are intrinsically temporal data (consider, e.g., the time when symptoms hold). Devising suitable representation formalisms to properly model such pieces of temporal information is a challenging task, for which several solutions have been proposed in the last years. Besides representation formalisms, temporal reasoning methodologies are also needed. Temporal abstraction is needed in order to infer abstract temporal data (as described in guideline action conditions) from "raw" timestamped patient data. Moreover, temporal constraint propagationis also needed, both at acquisition and at execution time. During acquisition, temporal constraint propagation is used to detect whether the temporal constraints in the guideline are consistent. At execution time, it is needed in order to check whether the actual time of execution of actions has respected the temporal constraints in the guideline, and to detect which are the next candidate actions to be executed, on the basis of the temporal constraints in the guideline. This chapter sketches some of the most important recent results about the above issues.

Applying artificial intelligence to clinical guidelines: the GLARE approach.

We present GLARE, a domain-independent system for acquiring, representing and executing clinical guidelines (GL). GLARE is characterized by the adoption of Artificial Intelligence (AI) techniques in the definition and implementation of the system. First of all, a high-level and user-friendly knowledge representation language has been designed. Second, a user-friendly acquisition tool, which provides expert physicians with various forms of help, has been implemented. Third, a tool for executing GL on a specific patient has been made available. At all the levels above, advanced AI techniques have been exploited, in order to enhance flexibility and user-friendliness and to provide decision support. Specifically, this chapter focuses on the methods we have developed in order to cope with (i) automatic resource-based adaptation of GL, (ii) representation and reasoning about temporal constraints in GL, (iii) decision making support, and (iv) model-based verification. We stress that, although we have devised such techniques within the GLARE project, they are mostly system-independent, so that they might be applied to other guideline management systems.

Automatic checking of the correctness of clinical guidelines in GLARE.

Representing clinical guidelines is a very complex knowledge-representation task, requiring a lot of expertise and efforts. Nevertheless, guideline representations often contain several kinds of errors. Therefore, checking the well-formedness and correctness of a guideline representation is an important task, which can be drastically improved with the adoption of computer programs. In this paper, we discuss the advanced facilities provided by the GLARE system to assist physicians to produce correct representations of clinical guidelines.