A community-of-practice-based evaluation methodology for knowledge intensive computational methods and its application to multimorbidity decision support.

OBJECTIVE: The study has dual objectives. Our first objective (1) is to develop a community-of-practice-based evaluation methodology for knowledge-intensive computational methods. We target a whitebox analysis of the computational methods to gain insight on their functional features and inner workings. In more detail, we aim to answer evaluation questions on (i) support offered by computational methods for functional features within the application domain; and (ii) in-depth characterizations of the underlying computational processes, models, data and knowledge of the computational methods. Our second objective (2) involves applying the evaluation methodology to answer questions (i) and (ii) for knowledge-intensive clinical decision support (CDS) methods, which operationalize clinical knowledge as computer interpretable guidelines (CIG); we focus on multimorbidity CIG-based clinical decision support (MGCDS) methods that target multimorbidity treatment plans. MATERIALS AND METHODS: Our methodology directly involves the research community of practice in (a) identifying functional features within the application domain; (b) defining exemplar case studies covering these features; and (c) solving the case studies using their developed computational methods-research groups detail their solutions and functional feature support in solution reports. Next, the study authors (d) perform a qualitative analysis of the solution reports, identifying and characterizing common themes (or dimensions) among the computational methods. This methodology is well suited to perform whitebox analysis, as it directly involves the respective developers in studying inner workings and feature support of computational methods. Moreover, the established evaluation parameters (e.g., features, case studies, themes) constitute a re-usable benchmark framework, which can be used to evaluate new computational methods as they are developed. We applied our community-of-practice-based evaluation methodology on MGCDS methods. RESULTS: Six research groups submitted comprehensive solution reports for the exemplar case studies. Solutions for two of these case studies were reported by all groups. We identified four evaluation dimensions: detection of adverse interactions, management strategy representation, implementation paradigms, and human-in-the-loop support. Based on our whitebox analysis, we present answers to the evaluation questions (i) and (ii) for MGCDS methods. DISCUSSION: The proposed evaluation methodology includes features of illuminative and comparison-based approaches; focusing on understanding rather than judging/scoring or identifying gaps in current methods. It involves answering evaluation questions with direct involvement of the research community of practice, who participate in setting up evaluation parameters and solving exemplar case studies. Our methodology was successfully applied to evaluate six MGCDS knowledge-intensive computational methods. We established that, while the evaluated methods provide a multifaceted set of solutions with different benefits and drawbacks, no single MGCDS method currently provides a comprehensive solution for MGCDS. CONCLUSION: We posit that our evaluation methodology, applied here to gain new insights into MGCDS, can be used to assess other types of knowledge-intensive computational methods and answer other types of evaluation questions. Our case studies can be accessed at our GitHub repository (https://github.com/william-vw/MGCDS).

Conformance analysis for comorbid patients in Answer Set Programming.

The treatment of comorbid patients is a hot problem in Medical Informatics, since the plain application of multiple Computer-Interpretable Guidelines (CIGs) can lead to interactions that are potentially dangerous for the patients. The specialized literature has mostly focused on the "a priori" or "execution-time" analysis of the interactions between multiple Computer-Interpretable Guidelines (CIGs), and/or CIG "merge". In this paper, we face a complementary problem, namely, the a posteriori analysis of the treatment of comorbid patients. Given the CIGs, the history of the status of the patient, and the log of the clinical actions executed on her, we try to explain the actions executed on the patient (i.e., the log) in terms of the actions recommended by the CIGs, of their potential interactions, and of the possible ways of managing each such interaction, pointing out (i) deviations from CIG recommendations not explained in terms of interaction management (if any) and (ii) unmanaged interactions (if any). Our approach is based on Answer Set Programming, and, to face realistic problems, devotes specific attention to the temporal dimension.

Managing proposals and evaluations of updates to medical knowledge: theory and applications.

The process of keeping up-to-date the medical knowledge stored in relational databases is of paramount importance. Since quality and reliability of medical knowledge are essential, in many cases physicians' proposals of updates must undergo experts' evaluation before possibly becoming effective. However, until now no theoretical framework has been provided in order to cope with this phenomenon in a principled and non-ad hoc way. Indeed, such a framework is important not only in the medical domain, but in all Wikipedia-like contexts in which evaluation of update proposals is required. In this paper we propose GPVM (General Proposal Vetting Model), a general model to cope with update proposal⧹evaluation in relational databases. GPVM extends the current theory of temporal relational databases and, in particular, BCDM - Bitemporal Conceptual Data Model - "consensus" model, providing a new data model, new operations to propose and accept⧹reject updates, and new algebraic operators to query proposals. The properties of GPVM are also studied. In particular, GPVM is a consistent extension of BCDM and it is reducible to it. These properties ensure consistency with most relational temporal database frameworks, facilitating implementation on top of current frameworks and interoperability with previous approaches.

Supporting physicians in the coordination of distributed execution of CIGs to treat comorbid patients.

Clinical Practice Guidelines (CPGs) encode the "best" medical practices to treat patients affected by a specific disease and are widely used in the medical practice. Starting from the '90s', several Computer-Interpretable Guideline (CIG) systems have been devised to provide physicians with CPG-based decision support. CPGs (and CIGs) are devoted to provide evidence-based recommendations for one specific disease. In order to support the treatment of patients affected by multiple diseases (i.e., comorbid patients), challenging additional tasks have to be addressed, such as (i) the detection of the interactions between CIG actions, (ii) their management, and, finally, (iii) the "merge" or conciliation of the CIGs. Several CIG approaches have been recently extended in order to face (at least one of) such challenging problems, and one of them is GLARE. However, besides the solutions to tasks (i)-(iii) above, the "run-time" support to physicians treating a comorbid patient requires additional capabilities, to support the distribution of the management of interactions and of the execution of CIGs among different physicians. In this paper, we propose a general framework, based on GLARE and GLARE-SSCPM, to provide such additional capabilities.

Supporting human interaction and human resources coordination in distributed clinical guidelines.

Clinical guidelines (GL) play an important role in medical practice: the one of optimizing the quality of patient care on the basis of the best and most recent evidence based medicine. In order to achieve this goal, the interaction between different actors, who cooperate in the execution of the same GL, is a crucial issue. As a matter of fact, in many cases (e.g. in chronic disease treatment) the GL execution requires that patient treatment is not performed/completed in the hospital, but is continued in different contexts (e.g. at home, or in the general practitioner's ambulatory), under the responsibility of different actors. In this situation, the correct interaction and communication between the actors themselves is critical for the quality of care, and human resources coordination is a key issue to be addressed by the managers of the involved healthcare service. In this paper we describe how computerized GL management can be extended in order to support such needs, and we illustrate our approach by means of a practical case study.

Towards an implicit treatment of periodically-repeated medical data.

Temporal information plays a crucial role in medicine, so that in Medical Informatics there is an increasing awareness that suitable database approaches are needed to store and support it. Specifically, a great amount of clinical data (e.g., therapeutic data) are periodically repeated. Although an explicit treatment is possible in most cases, it causes severe storage and disk I/O problems. In this paper, we propose an innovative approach to cope with periodic medical data in an implicit way. We propose a new data model, representing periodic data in a compact (implicit) way, which is a consistent extension of TSQL2 consensus approach. Then, we identify some important types of temporal queries, and present query answering algorithms to answer them. We also sketch a temporal relational algebra for our approach. Finally, we show experimentally that our approach outperforms current explicit approaches.

Supporting the distributed execution of clinical guidelines by multiple agents.

Clinical guidelines (GLs) are widely adopted in order to improve the quality of patient care, and to optimize it. To achieve such goals, their application on a specific patient usually requires the interventions of different agents, with different roles (e.g., physician, nurse), abilities (e.g., specialist in the treatment of alcohol-related problems) and contexts (e.g., many chronic patients may be treated at home). Additionally, the responsibility of the application of a guideline to a patient is usually retained by a physician, but delegation of responsibility (of the whole guideline, or of a part of it) is often used\required (e.g., delegation to a specialist), as well as the possibility, for a responsible, to select the executor of an action (e.g., a physician may retain the responsibility of an action, but delegate to a nurse its execution). To manage such phenomena, proper support to agent interaction and communication must be provided, providing agents with facilities for (1) treatment continuity (2) contextualization, (3) responsibility assignment and delegation (4) check of agent "appropriateness". In this paper we extend GLARE, a computerized GL management system, to support such needs. We illustrate our approach by means of a practical case study.

The temporal aspects of clinical guidelines.

Temporal aspects play a major role within clinical guidelines. Temporal issues arise when considering both guidelines per se, and the application of guidelines to specific patients. As a matter of fact, guidelines per se specify different diagnostic and\or therapeutic patterns, and temporal constraints on the intended times of execution of the actions they contain are an intrinsic part of guidelines themselves. Moreover, guidelines must be executed on the basis of patients' data, which are intrinsically temporal data (consider, e.g., the time when symptoms hold). Devising suitable representation formalisms to properly model such pieces of temporal information is a challenging task, for which several solutions have been proposed in the last years. Besides representation formalisms, temporal reasoning methodologies are also needed. Temporal abstraction is needed in order to infer abstract temporal data (as described in guideline action conditions) from "raw" timestamped patient data. Moreover, temporal constraint propagationis also needed, both at acquisition and at execution time. During acquisition, temporal constraint propagation is used to detect whether the temporal constraints in the guideline are consistent. At execution time, it is needed in order to check whether the actual time of execution of actions has respected the temporal constraints in the guideline, and to detect which are the next candidate actions to be executed, on the basis of the temporal constraints in the guideline. This chapter sketches some of the most important recent results about the above issues.

Applying artificial intelligence to clinical guidelines: the GLARE approach.

We present GLARE, a domain-independent system for acquiring, representing and executing clinical guidelines (GL). GLARE is characterized by the adoption of Artificial Intelligence (AI) techniques in the definition and implementation of the system. First of all, a high-level and user-friendly knowledge representation language has been designed. Second, a user-friendly acquisition tool, which provides expert physicians with various forms of help, has been implemented. Third, a tool for executing GL on a specific patient has been made available. At all the levels above, advanced AI techniques have been exploited, in order to enhance flexibility and user-friendliness and to provide decision support. Specifically, this chapter focuses on the methods we have developed in order to cope with (i) automatic resource-based adaptation of GL, (ii) representation and reasoning about temporal constraints in GL, (iii) decision making support, and (iv) model-based verification. We stress that, although we have devised such techniques within the GLARE project, they are mostly system-independent, so that they might be applied to other guideline management systems.

Representing and querying now-relative relational medical data.

Temporal information plays a crucial role in medicine. Patients' clinical records are intrinsically temporal. Thus, in Medical Informatics there is an increasing need to store, support and query temporal data (particularly in relational databases), in order, for instance, to supplement decision-support systems. In this paper, we show that current approaches to relational data have remarkable limitations in the treatment of "now-relative" data (i.e., data holding true at the current time). This can severely compromise their applicability in general, and specifically in the medical context, where "now-relative" data are essential to assess the current status of the patients. We propose a theoretically grounded and application-independent relational approach to cope with now-relative data (which can be paired, e.g., with different decision support systems) overcoming such limitations. We propose a new temporal relational representation, which is the first relational model coping with the temporal indeterminacy intrinsic in now-relative data. We also propose new temporal algebraic operators to query them, supporting the distinction between possible and necessary time, and Allen's temporal relations between data. We exemplify the impact of our approach, and study the theoretical and computational properties of the new representation and algebra.

Extending temporal databases to deal with telic/atelic medical data.

OBJECTIVE: In this paper, we aim at defining a general-purpose data model and query language coping with both "telic" and "atelic" medical data. BACKGROUND: In the area of Medical Informatics, there is an increasing realization that temporal information plays a crucial role, so that suitable database models and query languages are needed to store and support it. However, despite the wide range of approaches in the area, in this paper we show that a relevant class of medical data cannot be properly dealt with. METHODOLOGY: We first show that data models based on the "point-based" semantics, which is (implicitly or explicitly) assumed by the totality of temporal database approaches, have several limitations when dealing with "telic" data. We then propose a new model (based on the "interval-based" semantics) to cope with such data, and extend the query language accordingly. RESULTS: We propose a new three-sorted model and a query language to properly deal with both "telic" and "atelic" medical data (as well as non-temporal data). Our query language is flexible, since it allows one to switch from "atelic" to "telic" data, and vice versa. CONCLUSION: In this paper, we demonstrate the feasibility of a database approach copying with both telic and atelic data as needed in several (medical) applications.

Supporting therapy selection in computerized clinical guidelines by means of decision theory.

Supporting therapy selection is a fundamental task for a system for the computerized management of clinical guidelines (GL). The goal is particularly critical when no alternative is really better than the others, from a strictly clinical viewpoint. In these cases, decision theory appears to be a very suitable means to provide advice. In this paper, we describe how algorithms for calculating utility, and for evaluating the optimal policy, can be exploited to fit the GL management context.

Automatic treatment of temporal issues in clinical guidelines in the GLARE system.

Temporal constraints play a fundamental role in clinical guidelines. For example, temporal indeterminacy, constraints about duration, delays between actions and periodic repetitions of actions are essential in order to cope with clinical therapies. This paper proposes a computer-based approach in order to deal with temporal constraints in clinical guidelines. Specifically, it provides the possibility to represent such constraints and reason with them (i.e., perform inferences in the form of constraint propagation). We first propose a temporal representation formalism and two constraint propagation algorithms operating on it, and then we show how they can be exploited in order to provide clinical guideline systems with different temporal facilities. Our approach offers several advantages: for example, during the guideline acquisition phase, it enables to represent temporal constraints and to check their consistency; during the execution phase, it allows the physician to check the consistency between action execution-times and the constraints in the guidelines, and to provide query answering and temporal simulation facilities (e.g., when choosing among alternative paths in a guideline).

Automatic checking of the correctness of clinical guidelines in GLARE.

Representing clinical guidelines is a very complex knowledge-representation task, requiring a lot of expertise and efforts. Nevertheless, guideline representations often contain several kinds of errors. Therefore, checking the well-formedness and correctness of a guideline representation is an important task, which can be drastically improved with the adoption of computer programs. In this paper, we discuss the advanced facilities provided by the GLARE system to assist physicians to produce correct representations of clinical guidelines.

Temporal detection and analysis of guideline interactions.

BACKGROUND: Clinical practice guidelines (CPGs) are assuming a major role in the medical area, to grant the quality of medical assistance, supporting physicians with evidence-based information of interventions in the treatment of single pathologies. The treatment of patients affected by multiple diseases (comorbid patients) is one of the main challenges for the modern healthcare. It requires the development of new methodologies, supporting physicians in the treatment of interactions between CPGs. Several approaches have started to face such a challenging problem. However, they suffer from a substantial limitation: they do not take into account the temporal dimension. Indeed, practically speaking, interactions occur in time. For instance, the effects of two actions taken from different guidelines may potentially conflict, but practical conflicts happen only if the times of execution of such actions are such that their effects overlap in time. OBJECTIVES: We aim at devising a methodology to detect and analyse interactions between CPGs that considers the temporal dimension. METHODS: In this paper, we first extend our previous ontological model to deal with the fact that actions, goals, effects and interactions occur in time, and to model both qualitative and quantitative temporal constraints between them. Then, we identify different application scenarios, and, for each of them, we propose different types of facilities for user physicians, useful to support the temporal detection of interactions. RESULTS: We provide a modular approach in which different Artificial Intelligence temporal reasoning techniques, based on temporal constraint propagation, are widely exploited to provide users with such facilities. We applied our methodology to two cases of comorbidities, using simplified versions of CPGs. CONCLUSION: We propose an innovative approach to the detection and analysis of interactions between CPGs considering different sources of temporal information (CPGs, ontological knowledge and execution logs), which is the first one in the literature that takes into account the temporal issues, and accounts for different application scenarios.

Towards a comprehensive treatment of repetitions, periodicity and temporal constraints in clinical guidelines.

OBJECTIVE: In this paper, we define a principled approach to represent temporal constraints in clinical guidelines and to reason (i.e., perform inferences in the form of constraint propagation) on them. We consider different types of constraints, including composite and repeated actions, and propose different types of temporal functionalities (e.g., temporal consistency checking). BACKGROUND: Constraints about actions, durations, delays and periodic repetitions of actions are an intrinsic part of most clinical guidelines. Although several approaches provide expressive temporal formalisms, only few of them deal with the related temporal reasoning issues. METHODOLOGY: We first propose a temporal representation formalism and two temporal reasoning algorithms. Then, we consider the trade-off between the expressiveness of the formalism and the computational complexity of the algorithms, in order to devise a correct, complete and tractable approach. Finally, we show how the algorithms can be exploited to provide clinical guideline systems with different types of temporal facilities. RESULTS: Our approach offers several advantages. During the guideline acquisition phase, it enables to represent temporal constraints, and to check their consistency. In the execution phase, it checks the consistency between the execution times of the actions and the constraints in the guidelines, and provides query answering and simulation facilities.

META-GLARE: A meta-system for defining your own computer interpretable guideline system-Architecture and acquisition.

CONTEXT: Several different computer-assisted management systems of computer interpretable guidelines (CIGs) have been developed by the Artificial Intelligence in Medicine community. Each CIG system is characterized by a specific formalism to represent CIGs, and usually provides a manager to acquire, consult and execute them. Though there are several commonalities between most formalisms in the literature, each formalism has its own peculiarities. OBJECTIVE: The goal of our work is to provide a flexible support to the extension or definition of CIGs formalisms, and of their acquisition and execution engines. Instead of defining "yet another CIG formalism and its manager", we propose META-GLARE (META Guideline Acquisition, Representation, and Execution), a "meta"-system to define new CIG systems. METHOD AND MATERIALS: In this paper, META-GLARE, a meta-system to define new CIG systems, is presented. We try to capture the commonalities among current CIG approaches, by providing (i) a general manager for the acquisition, consultation and execution of hierarchical graphs (representing the control flow of actions in CIGs), parameterized over the types of nodes and of arcs constituting it, and (ii) a library of different elementary components of guidelines nodes (actions) and arcs, in which each type definition involves the specification of how objects of this type can be acquired, consulted and executed. We provide generality and flexibility, by allowing free aggregations of such elementary components to define new primitive node and arc types. RESULTS: We have drawn several experiments, in which we have used META-GLARE to build a CIG system (Experiment 1 in Section 8), or to extend it (Experiments 2 and 3). Such experiments show that META-GLARE provides a useful and easy-to-use support to such tasks. For instance, re-building the Guideline Acquisition, Representation, and Execution (GLARE) system using META-GLARE required less than one day (Experiment 1). CONCLUSIONS: META-GLARE is a meta-system for CIGs supporting fast prototyping. Since META-GLARE provides acquisition and execution engines that are parametric over the specific CIG formalism, it supports easy update and construction of CIG systems.

META-GLARE: a shell for CIG systems.

In the last twenty years, many different approaches to deal with Computer-Interpretable clinical Guidelines (CIGs) have been developed, each one proposing its own representation formalism (mostly based on the Task-Network Model) execution engine. We propose META-GLARE a shell for easily defining new CIG systems. Using META-GLARE, CIG system designers can easily define their own systems (basically by defining their representation language), with a minimal programming effort. META-GLARE is thus a flexible and powerful vehicle for research about CIGs, since it supports easy and fast prototyping of new CIG systems.

Towards a flexible integration of clinical guideline systems with medical ontologies and medical information systems.

The integration of a computer-based system dealing with clinical guidelines with a medical ontology can provide several advantages, including standardization and knowledge sharing. Furthermore, in order to operate in the clinical practice, guideline systems must also interact with the hospital databases to retrieve patients' data. Unfortunately, currently there seems not to be any "standard" consensus model either for the medical ontology or for (the conceptual structure of) patient databases (even if several interesting proposals have been carried out). In this paper we show how we are extending the GLARE guideline manager in order to strictly interact with both a medical ontology and a patient Database, in such a way that GLARE is not committed to any specific ontology and database (i.e., different ontologies and/or databases can be used).

The GLARE approach to clinical guidelines: main features.

In this paper, we present GLARE, a domain-independent prototypical system for acquiring, representing and executing clinical guidelines. GLARE has been built within a 7-year project with Azienda Ospedaliera San Giovanni Battista in Turin (one of the largest hospitals in Italy) and has been successfully tested on clinical guidelines in different domains, including bladder cancer, reflux esophagitis, and heart failure. GLARE is characterized by the adoption of advanced Artificial Intelligence (AI) techniques, to support medical decision making and to manage temporal knowledge.