Harmonization of ICF Body Structures and ICD-11 Anatomic Detail: One foundation for multiple classifications.

The Family of International Classifications of the World Health Organization (WHO-FIC) currently includes three reference classifications, namely International Classification of Diseases (ICD), International Classification of Functioning, Disability, and Health (ICF), and International Classification of Health Interventions (ICHI). Recently, the three classifications have been incorporated into a single WHO-FIC Foundation that serves as the repository of all concepts in the classifications. Each classification serves a specific classification need. However, they share some common concepts that are present, in different forms, in two or all of them. For the WHO-FIC Foundation to be a logically consistent repository without duplicates, these common concepts must be reconciled. One important set of shared concepts is the representation of human anatomy entities, which are not always modeled in the same way and with the same level of detail. To understand the relationships among the three anatomical representations, an effort is needed to compare them, identifying common areas, gaps, and compatible and incompatible modeling. The work presented here contributes to this effort, focusing on the anatomy representations in ICF and ICD-11. For this aim, three experts were asked to identify, for each entity in the ICF Body Structures, one or more entities in the ICD-11 Anatomic Detail that could be considered identical, broader or narrower. To do this, they used a specifically developed web application, which also automatically identified the most obvious equivalences. A total of 631 maps were independently identified by the three mappers for 218 ICF Body Structures, with an interobserver agreement of 93.5%. Together with 113 maps identified by the software, they were then consolidated into 434 relations. The results highlight some differences between the two classifications: in general, ICF is less detailed than ICD-11; ICF favors lumping of structures; in very few cases, the two classifications follow different anatomic models. For these issues, solutions have to be found that are compliant with the WHO approach to classification modeling and maintenance.

A community-of-practice-based evaluation methodology for knowledge intensive computational methods and its application to multimorbidity decision support.

OBJECTIVE: The study has dual objectives. Our first objective (1) is to develop a community-of-practice-based evaluation methodology for knowledge-intensive computational methods. We target a whitebox analysis of the computational methods to gain insight on their functional features and inner workings. In more detail, we aim to answer evaluation questions on (i) support offered by computational methods for functional features within the application domain; and (ii) in-depth characterizations of the underlying computational processes, models, data and knowledge of the computational methods. Our second objective (2) involves applying the evaluation methodology to answer questions (i) and (ii) for knowledge-intensive clinical decision support (CDS) methods, which operationalize clinical knowledge as computer interpretable guidelines (CIG); we focus on multimorbidity CIG-based clinical decision support (MGCDS) methods that target multimorbidity treatment plans. MATERIALS AND METHODS: Our methodology directly involves the research community of practice in (a) identifying functional features within the application domain; (b) defining exemplar case studies covering these features; and (c) solving the case studies using their developed computational methods-research groups detail their solutions and functional feature support in solution reports. Next, the study authors (d) perform a qualitative analysis of the solution reports, identifying and characterizing common themes (or dimensions) among the computational methods. This methodology is well suited to perform whitebox analysis, as it directly involves the respective developers in studying inner workings and feature support of computational methods. Moreover, the established evaluation parameters (e.g., features, case studies, themes) constitute a re-usable benchmark framework, which can be used to evaluate new computational methods as they are developed. We applied our community-of-practice-based evaluation methodology on MGCDS methods. RESULTS: Six research groups submitted comprehensive solution reports for the exemplar case studies. Solutions for two of these case studies were reported by all groups. We identified four evaluation dimensions: detection of adverse interactions, management strategy representation, implementation paradigms, and human-in-the-loop support. Based on our whitebox analysis, we present answers to the evaluation questions (i) and (ii) for MGCDS methods. DISCUSSION: The proposed evaluation methodology includes features of illuminative and comparison-based approaches; focusing on understanding rather than judging/scoring or identifying gaps in current methods. It involves answering evaluation questions with direct involvement of the research community of practice, who participate in setting up evaluation parameters and solving exemplar case studies. Our methodology was successfully applied to evaluate six MGCDS knowledge-intensive computational methods. We established that, while the evaluated methods provide a multifaceted set of solutions with different benefits and drawbacks, no single MGCDS method currently provides a comprehensive solution for MGCDS. CONCLUSION: We posit that our evaluation methodology, applied here to gain new insights into MGCDS, can be used to assess other types of knowledge-intensive computational methods and answer other types of evaluation questions. Our case studies can be accessed at our GitHub repository (https://github.com/william-vw/MGCDS).

Structural Patterns in Chronic Disease Clinical Practice Guidelines Formalized for Clinical Decision Support.

Making recommendations from clinical practice guidelines (CPGs) computable for clinical decision support (CDS) has typically been a laborious and costly process. Identifying domain-specific regularities helps clinicians and knowledge engineers conceptualize, extract, and encode evidence-based recommendations. Based on our work to provide complex CDS in the management of multiple chronic diseases, we propose nine chronic disease CPG structural patterns, discuss considerations in representing the necessary knowledge, and illustrate them with the solutions that our CDS system provides.

Towards a framework for comparing functionalities of multimorbidity clinical decision support: A literature-based feature set and benchmark cases.

Multimorbidity, the coexistence of two or more health conditions, has become more prevalent as mortality rates in many countries have declined and their populations have aged. Multimorbidity presents significant difficulties for Clinical Decision Support Systems (CDSS), particularly in cases where recommendations from relevant clinical guidelines offer conflicting advice. A number of research groups are developing computer-interpretable guideline (CIG) modeling formalisms that integrate recommendations from multiple Clinical Practice Guidelines (CPGs) for knowledge-based multimorbidity decision support. In this paper we describe work towards the development of a framework for comparing the different approaches to multimorbidity CIG-based clinical decision support (MGCDS). We present (1) a set of features for MGCDS, which were derived using a literature review and evaluated by physicians using a survey, and (2) a set of benchmarking case studies, which illustrate the clinical application of these features. This work represents the first necessary step in a broader research program aimed at the development of a benchmark framework that allows for standardized and comparable MGCDS evaluations, which will facilitate the assessment of functionalities of MGCDS, as well as highlight important gaps in the state-of-the-art. We also outline our future work on developing the framework, specifically, (3) a standard for reporting MGCDS solutions for the benchmark case studies, and (4) criteria for evaluating these MGCDS solutions. We plan to conduct a large-scale comparison study of existing MGCDS based on the comparative framework.

Towards a goal-oriented methodology for clinical-guideline-based management recommendations for patients with multimorbidity: GoCom and its preliminary evaluation.

Patients with chronic multimorbidity are becoming more common as life expectancy increases, making it necessary for physicians to develop complex management plans. We are looking at the patient management process as a goal-attainment problem. Hence, our aim is to develop a goal-oriented methodology for providing decision support for managing patients with multimorbidity continuously, as the patient's health state is progressing and new goals arise (e.g., treat ulcer, prevent osteoporosis). Our methodology allows us to detect and mitigate inconsistencies among guideline recommendations stemming from multiple clinical guidelines, while consulting medical ontologies and terminologies and relying on patient information standards. This methodology and its implementation as a decision-support system, called GoCom, starts with computer-interpretable clinical guidelines (CIGs) for single problems that are formalized using the PROforma CIG language. We previously published the architecture of the system as well as a CIG elicitation guide for enriching PROforma tasks with properties referring to vocabulary codes of goals and physiological effects of management plans. In this paper, we provide a formalization of the conceptual model of GoCom that generates, for each morbidity of the patient, a patient-specific goal tree that results from the PROforma engine's enactment of the CIG with the patient's data. We also present the "Controller" algorithm that drives the GoCom system. Given a new problem that a patient develops, the Controller detects inconsistencies among goals pertaining to different comorbid problems and consults the CIGs to generate alternative non-conflicted and goal-oriented management plans that address the multiple goals simultaneously. In this stage of our research, the inconsistencies that can be detected are of two types - starting vs. stopping medications that belong to the same medication class hierarchy, and detecting opposing physiological effect goals that are specified in concurrent CIGs (e.g., decreased blood pressure vs. increased blood pressure). However, the design of GoCom is modular and generic and allows the future introduction of additional interaction detection and mitigation strategies. Moreover, GoCom generates explanations of the alternative non-conflicted management plans, based on recommendations stemming from the clinical guidelines and reasoning patterns. GoCom's functionality was evaluated using three cases of multimorbidity interactions that were checked by our three clinicians. Usefulness was evaluated with two studies. The first evaluation was a pilot study with ten 6th year medical students and the second evaluation was done with 27 6th medical students and interns. The participants solved complex realistic cases of multimorbidity patients: with and without decision-support, two cases in the first evaluation and 6 cases in the second evaluation. Use of GoCom increased completeness of the patient management plans produced by the medical students from 0.44 to 0.71 (P-value of 0.0005) in the first evaluation, and from 0.31 to 0.78 (P-value < 0.0001) in the second evaluation. Correctness in the first evaluation was very high with (0.98) or without the system (0.91), with non-significant difference (P-value ≥ 0.17). In the second evaluation, use of GoCom increased correctness from 0.68 to 0.83 (P-value of 0.001). In addition, GoCom's explanation and visualization were perceived as useful by the vast majority of participants. While GoCom's detection of goal interactions is currently limited to detection of starting vs. stopping the same medication or medication subclasses and detecting conflicting physiological effects of concurrent medications, the evaluation demonstrated potential of the system for improving clinical decision-making for multimorbidity patients.

Toward a Harmonized WHO Family of International Classifications Content Model.

An overarching WHO-FIC Content Model will allow uniform modeling of classifications in the WHO Family of International Classifications (WHO-FIC) and promote their joint use. We provide an initial conceptualization of such a model.

Selecting Test Cases from the Electronic Health Record for Software Testing of Knowledge-Based Clinical Decision Support Systems.

Software testing of knowledge-based clinical decision support systems is challenging, labor intensive, and expensive; yet, testing is necessary since clinical applications have heightened consequences. Thoughtful test case selection improves testing coverage while minimizing testing burden. ATHENA-CDS is a knowledge-based system that provides guideline-based recommendations for chronic medical conditions. Using the ATHENA-CDS diabetes knowledgebase, we demonstrate a generalizable approach for selecting test cases using rules/ filters to create a set of paths that mimics the system's logic. Test cases are allocated to paths using a proportion heuristic. Using data from the electronic health record, we found 1,086 cases with glycemic control above target goals. We created a total of 48 filters and 50 unique system paths, which were used to allocate 200 test cases. We show that our method generates a comprehensive set of test cases that provides adequate coverage for the testing of a knowledge-based CDS.

Goal-driven management of interacting clinical guidelines for multimorbidity patients.

Computer-interpretable guidelines (CIGs) are based on clinical practice guidelines, which typically address a single morbidity. However, most of the aging population suffers from multiple morbidities. Currently, there is no demonstrated effective mechanism that integrates recommendations from multiple CIGs. We are developing a goal-based method that utilizes knowledge of drugs' physiological effects and therapeutic usage to combine knowledge from CIGs. It incrementally detects interactions and plans non-contradicting therapies. Our algorithm uses patterns to check consistency and respond to events, including data enquiries, diagnoses, adverse events, recommended medications, tests, and goals. Our method utilizes existing standards and CIG tools, including the Fast Healthcare Interoperability Resources (FHIR) patient data model, SNOMED-CT, and the PROforma CIG formalism with its Alium knowledge-engineering environment and PROforma enactment engine. We demonstrate our approach using a case study involving two clinical guidelines with templates for responding to a new goal and to a medication request that causes an inconsistency which can be automatically detected and resolved based on the knowledge of the two CIGs.

An ontology-driven tool for structured data acquisition using Web forms.

BACKGROUND: Structured data acquisition is a common task that is widely performed in biomedicine. However, current solutions for this task are far from providing a means to structure data in such a way that it can be automatically employed in decision making (e.g., in our example application domain of clinical functional assessment, for determining eligibility for disability benefits) based on conclusions derived from acquired data (e.g., assessment of impaired motor function). To use data in these settings, we need it structured in a way that can be exploited by automated reasoning systems, for instance, in the Web Ontology Language (OWL); the de facto ontology language for the Web. RESULTS: We tackle the problem of generating Web-based assessment forms from OWL ontologies, and aggregating input gathered through these forms as an ontology of "semantically-enriched" form data that can be queried using an RDF query language, such as SPARQL. We developed an ontology-based structured data acquisition system, which we present through its specific application to the clinical functional assessment domain. We found that data gathered through our system is highly amenable to automatic analysis using queries. CONCLUSIONS: We demonstrated how ontologies can be used to help structuring Web-based forms and to semantically enrich the data elements of the acquired structured data. The ontologies associated with the enriched data elements enable automated inferences and provide a rich vocabulary for performing queries.

High-Risk Drug-Drug Interactions Between Clinical Practice Guidelines for Management of Chronic Conditions.

Clinicians and clinical decision-support systems often follow pharmacotherapy recommendations for patients based on clinical practice guidelines (CPGs). In multimorbid patients, these recommendations can potentially have clinically significant drug-drug interactions (DDIs). In this study, we describe and validate a method for programmatically detecting DDIs among CPG recommendations. The system extracts pharmacotherapy intervention recommendations from narrative CPGs, normalizes the terms, creates a mapping of drugs and drug classes, and then identifies occurrences of DDIs between CPG pairs. We used this system to analyze 75 CPGs written by authoring entities in the United States that discuss outpatient management of common chronic diseases. Using a reference list of high-risk DDIs, we identified 2198 of these DDIs in 638 CPG pairs (46 unique CPGs). Only 9 high-risk DDIs were discussed by both CPGs in a pairing. In 69 of the pairings, neither CPG had a pharmacologic reference or a warning of the possibility of a DDI.

Use of ontology structure and Bayesian models to aid the crowdsourcing of ICD-11 sanctioning rules.

The International Classification of Diseases (ICD) is the de facto standard international classification for mortality reporting and for many epidemiological, clinical, and financial use cases. The next version of ICD, ICD-11, will be submitted for approval by the World Health Assembly in 2018. Unlike previous versions of ICD, where coders mostly select single codes from pre-enumerated disease and disorder codes, ICD-11 coding will allow extensive use of multiple codes to give more detailed disease descriptions. For example, "severe malignant neoplasms of left breast" may be coded using the combination of a "stem code" (e.g., code for malignant neoplasms of breast) with a variety of "extension codes" (e.g., codes for laterality and severity). The use of multiple codes (a process called post-coordination), while avoiding the pitfall of having to pre-enumerate vast number of possible disease and qualifier combinations, risks the creation of meaningless expressions that combine stem codes with inappropriate qualifiers. To prevent that from happening, "sanctioning rules" that define legal combinations are necessary. In this work, we developed a crowdsourcing method for obtaining sanctioning rules for the post-coordination of concepts in ICD-11. Our method utilized the hierarchical structures in the domain to improve the accuracy of the sanctioning rules and to lower the crowdsourcing cost. We used Bayesian networks to model crowd workers' skills, the accuracy of their responses, and our confidence in the acquired sanctioning rules. We applied reinforcement learning to develop an agent that constantly adjusted the confidence cutoffs during the crowdsourcing process to maximize the overall quality of sanctioning rules under a fixed budget. Finally, we performed formative evaluations using a skin-disease branch of the draft ICD-11 and demonstrated that the crowd-sourced sanctioning rules replicated those defined by an expert dermatologist with high precision and recall. This work demonstrated that a crowdsourcing approach could offer a reasonably efficient method for generating a first draft of sanctioning rules that subject matter experts could verify and edit, thus relieving them of the tedium and cost of formulating the initial set of rules.

Test Case Selection in Pre-Deployment Testing of Complex Clinical Decision Support Systems.

Clinical decision support (CDS) systems with complex logic are being developed. Ensuring the quality of CDS is imperative, but there is no consensus on testing standards. We tested ATHENA-HTN CDS after encoding updated hypertension guidelines into the system. A logic flow and a complexity analysis of the encoding were performed to guide testing. 100 test cases were selected to test the major pathways in the CDS logic flow, and the effectiveness of the testing was analyzed. The encoding contained 26 decision points and 3120 possible output combinations. The 100 cases selected tested all of the major pathways in the logic, but only 1% of the possible output combinations. Test case selection is one of the most challenging aspects in CDS testing and has a major impact on testing coverage. A test selection strategy should take into account the complexity of the system, identification of major logic pathways, and available resources.

Automating Guidelines for Clinical Decision Support: Knowledge Engineering and Implementation.

As utilization of clinical decision support (CDS) increases, it is important to continue the development and refinement of methods to accurately translate the intention of clinical practice guidelines (CPG) into a computable form. In this study, we validate and extend the 13 steps that Shiffman et al.5 identified for translating CPG knowledge for use in CDS. During an implementation project of ATHENA-CDS, we encoded complex CPG recommendations for five common chronic conditions for integration into an existing clinical dashboard. Major decisions made during the implementation process were recorded and categorized according to the 13 steps. During the implementation period, we categorized 119 decisions and identified 8 new categories required to complete the project. We provide details on an updated model that outlines all of the steps used to translate CPG knowledge into a CDS integrated with existing health information technology.

Automating Performance Measures and Clinical Practice Guidelines: Differences and Complementarities.

Through close analysis of two pairs of systems that implement the automated evaluation of performance measures (PMs) and guideline-based clinical decision support (CDS), we contrast differences in their knowledge encoding and necessary changes to a CDS system that provides management recommendations for patients failing performance measures. We trace the sources of differences to the implementation environments and goals of PMs and CDS.

Summarizing and visualizing structural changes during the evolution of biomedical ontologies using a Diff Abstraction Network.

Biomedical ontologies are a critical component in biomedical research and practice. As an ontology evolves, its structure and content change in response to additions, deletions and updates. When editing a biomedical ontology, small local updates may affect large portions of the ontology, leading to unintended and potentially erroneous changes. Such unwanted side effects often go unnoticed since biomedical ontologies are large and complex knowledge structures. Abstraction networks, which provide compact summaries of an ontology's content and structure, have been used to uncover structural irregularities, inconsistencies and errors in ontologies. In this paper, we introduce Diff Abstraction Networks ("Diff AbNs"), compact networks that summarize and visualize global structural changes due to ontology editing operations that result in a new ontology release. A Diff AbN can be used to support curators in identifying unintended and unwanted ontology changes. The derivation of two Diff AbNs, the Diff Area Taxonomy and the Diff Partial-area Taxonomy, is explained and Diff Partial-area Taxonomies are derived and analyzed for the Ontology of Clinical Research, Sleep Domain Ontology, and eagle-i Research Resource Ontology. Diff Taxonomy usage for identifying unintended erroneous consequences of quality assurance and ontology merging are demonstrated.

A Method to Compare ICF and SNOMED CT for Coverage of U.S. Social Security Administration's Disability Listing Criteria.

We developed a method to evaluate the extent to which the International Classification of Function, Disability, and Health (ICF) and SNOMED CT cover concepts used in the disability listing criteria of the U.S. Social Security Administration's "Blue Book." First we decomposed the criteria into their constituent concepts and relationships. We defined different types of mappings and manually mapped the recognized concepts and relationships to either ICF or SNOMED CT. We defined various metrics for measuring the coverage of each terminology, taking into account the effects of inexact matches and frequency of occurrence. We validated our method by mapping the terms in the disability criteria of Adult Listings, Chapter 12 (Mental Disorders). SNOMED CT dominates ICF in almost all the metrics that we have computed. The method is applicable for determining any terminology's coverage of eligibility criteria.

Using a Clinical Knowledge Base to Assess Comorbidity Interrelatedness Among Patients with Multiple Chronic Conditions.

Decision support tools increasingly integrate clinical knowledge such as medication indications and contraindications with electronic health record (EHR) data to support clinical care and patient safety. The availability of this encoded information and patient data provides an opportunity to develop measures of clinical decision complexity that may be of value for quality improvement and research efforts. We investigated the feasibility of using encoded clinical knowledge and EHR data to develop a measure of comorbidity interrelatedness (the degree to which patients' co-occurring conditions interact to generate clinical complexity). Using a common clinical scenario-decisions about blood pressure medications in patients with hypertension-we quantified comorbidity interrelatedness by calculating the number of indications and contraindications to blood pressure medications that are generated by patients' comorbidities (e.g., diabetes, gout, depression). We examined properties of comorbidity interrelatedness using data from a decision support system for hypertension in the Veterans Affairs Health Care System.

The Ontology of Clinical Research (OCRe): an informatics foundation for the science of clinical research.

To date, the scientific process for generating, interpreting, and applying knowledge has received less informatics attention than operational processes for conducting clinical studies. The activities of these scientific processes - the science of clinical research - are centered on the study protocol, which is the abstract representation of the scientific design of a clinical study. The Ontology of Clinical Research (OCRe) is an OWL 2 model of the entities and relationships of study design protocols for the purpose of computationally supporting the design and analysis of human studies. OCRe's modeling is independent of any specific study design or clinical domain. It includes a study design typology and a specialized module called ERGO Annotation for capturing the meaning of eligibility criteria. In this paper, we describe the key informatics use cases of each phase of a study's scientific lifecycle, present OCRe and the principles behind its modeling, and describe applications of OCRe and associated technologies to a range of clinical research use cases. OCRe captures the central semantics that underlies the scientific processes of clinical research and can serve as an informatics foundation for supporting the entire range of knowledge activities that constitute the science of clinical research.

EliXR-TIME: A Temporal Knowledge Representation for Clinical Research Eligibility Criteria.

Effective clinical text processing requires accurate extraction and representation of temporal expressions. Multiple temporal information extraction models were developed but a similar need for extracting temporal expressions in eligibility criteria (e.g., for eligibility determination) remains. We identified the temporal knowledge representation requirements of eligibility criteria by reviewing 100 temporal criteria. We developed EliXR-TIME, a frame-based representation designed to support semantic annotation for temporal expressions in eligibility criteria by reusing applicable classes from well-known clinical temporal knowledge representations. We used EliXR-TIME to analyze a training set of 50 new temporal eligibility criteria. We evaluated EliXR-TIME using an additional random sample of 20 eligibility criteria with temporal expressions that have no overlap with the training data, yielding 92.7% (76 / 82) inter-coder agreement on sentence chunking and 72% (72 / 100) agreement on semantic annotation. We conclude that this knowledge representation can facilitate semantic annotation of the temporal expressions in eligibility criteria.