The Cooperation Databank: Machine-Readable Science Accelerates Research Synthesis.

Publishing studies using standardized, machine-readable formats will enable machines to perform meta-analyses on demand. To build a semantically enhanced technology that embodies these functions, we developed the Cooperation Databank (CoDa)-a databank that contains 2,636 studies on human cooperation (1958-2017) conducted in 78 societies involving 356,283 participants. Experts annotated these studies along 312 variables, including the quantitative results (13,959 effects). We designed an ontology that defines and relates concepts in cooperation research and that can represent the relationships between results of correlational and experimental studies. We have created a research platform that, given the data set, enables users to retrieve studies that test the relation of variables with cooperation, visualize these study results, and perform (a) meta-analyses, (b) metaregressions, (c) estimates of publication bias, and (d) statistical power analyses for future studies. We leveraged the data set with visualization tools that allow users to explore the ontology of concepts in cooperation research and to plot a citation network of the history of studies. CoDa offers a vision of how publishing studies in a machine-readable format can establish institutions and tools that improve scientific practices and knowledge.

Machine Learning Explainability in Breast Cancer Survival.

Machine Learning (ML) can improve the diagnosis, treatment decisions, and understanding of cancer. However, the low explainability of how "black box" ML methods produce their output hinders their clinical adoption. In this paper, we used data from the Netherlands Cancer Registry to generate a ML-based model to predict 10-year overall survival of breast cancer patients. Then, we used Local Interpretable Model-Agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP) to interpret the model's predictions. We found that, overall, LIME and SHAP tend to be consistent when explaining the contribution of different features. Nevertheless, the feature ranges where they have a mismatch can also be of interest, since they can help us identifying "turning points" where features go from favoring survived to favoring deceased (or vice versa). Explainability techniques can pave the way for better acceptance of ML techniques. However, their evaluation and translation to real-life scenarios need to be researched further.

Ten years of knowledge representation for health care (2009-2018): Topics, trends, and challenges.

BACKGROUND: In the last ten years, the international workshop on knowledge representation for health care (KR4HC) has hosted outstanding contributions of the artificial intelligence in medicine community pertaining to the formalization and representation of medical knowledge for supporting clinical care. Contributions regarding modeling languages, technologies and methodologies to produce these models, their incorporation into medical decision support systems, and practical applications in concrete medical settings have been the main contributions and the basis to define the evolution of this field across Europe and worldwide. OBJECTIVES: Carry out a review of the papers accepted in KR4HC in the 2009-2018 decade, analyze and characterize the topics and trends within this field, and identify challenges for the evolution of the area in the near future. METHODS: We reviewed the title, the abstract, and the keywords of the 112 papers that were accepted to the workshop, identified the medical and technological topics involved in these works, provided a classification of these papers in medical and technological perspectives and obtained the timeline of these topics in order to determine interest growths and declines. The experience of the authors in the field and the evidences after the review were the basis to propose a list of challenges of knowledge representation in health care for the future. RESULTS: The most generic knowledge representation methods are ontologies (31%), semantic web related formalisms (26%), decision tables and rules (19%), logic (14%), and probabilistic models (10%). From a medical informatics perspective, knowledge is mainly represented as computer interpretable clinical guidelines (43%), medical domain ontologies (26%), and electronic health care records (22%). Within the knowledge lifecycle, contributions are found in knowledge generation (38%), knowledge specification (24%), exception detection and management (12%), knowledge enactment (8%), temporal knowledge and reasoning (7%), and knowledge sharing and maintenance (7%). The clinical emphasis of knowledge is mainly related to clinical treatments (27%), diagnosis (13%), clinical quality indicators (13%), and guideline integration for multimorbid patients (12%). According to the level of development of the works presented, we distinguished four maturity levels: formal (22%), implementation (52%), testing (13%), and deployment (2%) levels. Some papers described technologies for specific clinical issues or diseases, mainly cancer (22%) and diseases of the circulatory system (20%). Chronicity and comorbidity were present in 10% and 8% of the papers, respectively. CONCLUSIONS: KR4HC is a stable community, still active after ten years. A persistent focus has been knowledge representation, with an emphasis on semantic-web ontologies and on clinical-guideline based decision-support. Among others, two topics receive growing attention: integration of computer-interpretable guideline knowledge for the management of multimorbidity patients, and patient empowerment and patient-centric care.

Analyzing interactions on combining multiple clinical guidelines.

Accounting for patients with multiple health conditions is a complex task that requires analysing potential interactions among recommendations meant to address each condition. Although some approaches have been proposed to address this issue, important features still require more investigation, such as (re)usability and scalability. To this end, this paper presents an approach that relies on reusable rules for detecting interactions among recommendations coming from various guidelines. It extends a previously proposed knowledge representation model (TMR) to enhance the detection of interactions and it provides a systematic analysis of relevant interactions in the context of multimorbidity. The approach is evaluated in a case study on rehabilitation of breast cancer patients, developed in collaboration with experts. The results are considered promising to support the experts in this task.

Identification of patients at risk for colorectal cancer in primary care: an explorative study with routine healthcare data.

BACKGROUND: Early diagnosis of colorectal cancer (CRC) is likely to reduce burden of disease and improve treatment success. Estimation of the individual patient risk for CRC diagnostic determinants in a primary care setting has not been very successful as yet. The aim of our study is to improve prediction of CRC in patients selected for colonoscopy in the primary healthcare setting using readily available routine healthcare data. PATIENTS AND METHODS: A cross-sectional study was carried out in the Julius General Practitioners' Network database. Patients referred for colonoscopy by their general practitioner (GP) between 2007 and 2012 were selected. We evaluated the association between long-term registered patient characteristics, symptoms and conditions, and colonoscopy test results with multivariable logistic regression. RESULTS: Two per cent (2787/140 000) of the patients between 30 and 85 years were found to be newly referred for colonoscopy by their GP, of whom 57 (2%) were diagnosed with CRC. Age 50 years or over, hypertension and the absence of preceding consultations for abdominal pain were independent predictors for CRC and/or high-risk adenomas, with an area under the curve of 0.65. CONCLUSION: Three factors in routine care data combined might prove valuable in future strategies to improve the prediction of CRC risk in primary care. Improvement in quality and availability of routine care data for research and risk stratification is needed to optimize its usability for prediction purposes in daily practice. IMPACT: Only referring patients at the highest risk for colonoscopy by the GP could decrease superfluous colonoscopies.

Enhancing reuse of structured eligibility criteria and supporting their relaxation.

Patient recruitment is one of the most important barriers to successful completion of clinical trials and thus to obtaining evidence about new methods for prevention, diagnostics and treatment. The reason is that recruitment is effort consuming. It requires the identification of candidate patients for the trial (the population under study), and verifying for each patient whether the eligibility criteria are met. The work we describe in this paper aims to support the comparison of population under study in different trials, and the design of eligibility criteria for new trials. We do this by introducing structured eligibility criteria, that enhance reuse of criteria across trials. We developed a method that allows for automated structuring of criteria from text. Additionally, structured eligibility criteria allow us to propose suggestions for relaxation of criteria to remove potentially unnecessarily restrictive conditions. We thereby increase the recruitment potential and generalizability of a trial. Our method for automated structuring of criteria enables us to identify related conditions and to compare their restrictiveness. The comparison is based on the general meaning of criteria, comprised of commonly occurring contextual patterns, medical concepts and constraining values. These are automatically identified using our pattern detection algorithm, state of the art ontology annotators and semantic taggers. The comparison uses predefined relations between the patterns, concept equivalences defined in medical ontologies, and threshold values. The result is a library of structured eligibility criteria which can be browsed using fine grained queries. Furthermore, we developed visualizations for the library that enable intuitive navigation of relations between trials, criteria and concepts. These visualizations expose interesting co-occurrences and correlations, potentially enhancing meta-research. The method for criteria structuring processes only certain types of criteria, which results in low recall of the method (18%) but a high precision for the relations we identify between the criteria (94%). Analysis of the approach from the medical perspective revealed that the approach can be beneficial for supporting trial design, though more research is needed.

Influence of data quality on computed Dutch hospital quality indicators: a case study in colorectal cancer surgery.

BACKGROUND: Our study aims to assess the influence of data quality on computed Dutch hospital quality indicators, and whether colorectal cancer surgery indicators can be computed reliably based on routinely recorded data from an electronic medical record (EMR). METHODS: Cross-sectional study in a department of gastrointestinal oncology in a university hospital, in which a set of 10 indicators is computed (1) based on data abstracted manually for the national quality register Dutch Surgical Colorectal Audit (DSCA) as reference standard and (2) based on routinely collected data from an EMR. All 75 patients for whom data has been submitted to the DSCA for the reporting year 2011 and all 79 patients who underwent a resection of a primary colorectal carcinoma in 2011 according to structured data in the EMR were included. Comparison of results, investigating the causes for any differences based on data quality analysis. Main outcome measures are the computability of quality indicators, absolute percentages of indicator results, data quality in terms of availability in a structured format, completeness and correctness. RESULTS: All indicators were fully computable based on the DSCA dataset, but only three based on EMR data, two of which were percentages. For both percentages, the difference in proportions computed based on the two datasets was significant.All required data items were available in a structured format in the DSCA dataset. Their average completeness was 86%, while the average completeness of these items in the EMR was 50%. Their average correctness was 87%. CONCLUSIONS: Our study showed that data quality can significantly influence indicator results, and that our EMR data was not suitable to reliably compute quality indicators. EMRs should be designed in a way so that the data required for audits can be entered directly in a structured and coded format.

Formalization and computation of quality measures based on electronic medical records.

OBJECTIVE: Ambiguous definitions of quality measures in natural language impede their automated computability and also the reproducibility, validity, timeliness, traceability, comparability, and interpretability of computed results. Therefore, quality measures should be formalized before their release. We have previously developed and successfully applied a method for clinical indicator formalization (CLIF). The objective of our present study is to test whether CLIF is generalizable--that is, applicable to a large set of heterogeneous measures of different types and from various domains. MATERIALS AND METHODS: We formalized the entire set of 159 Dutch quality measures for general practice, which contains structure, process, and outcome measures and covers seven domains. We relied on a web-based tool to facilitate the application of our method. Subsequently, we computed the measures on the basis of a large database of real patient data. RESULTS: Our CLIF method enabled us to fully formalize 100% of the measures. Owing to missing functionality, the accompanying tool could support full formalization of only 86% of the quality measures into Structured Query Language (SQL) queries. The remaining 14% of the measures required manual application of our CLIF method by directly translating the respective criteria into SQL. The results obtained by computing the measures show a strong correlation with results computed independently by two other parties. CONCLUSIONS: The CLIF method covers all quality measures after having been extended by an additional step. Our web tool requires further refinement for CLIF to be applied completely automatically. We therefore conclude that CLIF is sufficiently generalizable to be able to formalize the entire set of Dutch quality measures for general practice.

Barriers to the reuse of routinely recorded clinical data: a field report.

Today, clinical data is routinely recorded in vast amounts, but its reuse can be challenging. A secondary use that should ideally be based on previously collected clinical data is the computation of clinical quality indicators. In the present study, we attempted to retrieve all data from our hospital that is required to compute a set of quality indicators in the domain of colorectal cancer surgery. We categorised the barriers that we encountered in the scope of this project according to an existing framework, and provide recommendations on how to prevent or surmount these barriers. Assuming that our case is not unique, these recommendations might be applicable for the design, evaluation and optimisation of Electronic Health Records.

Automated generation of patient-tailored electronic care pathways by translating computer-interpretable guidelines into hierarchical task networks.

OBJECTIVE: This paper describes a methodology which enables computer-aided support for the planning, visualization and execution of personalized patient treatments in a specific healthcare process, taking into account complex temporal constraints and the allocation of institutional resources. To this end, a translation from a time-annotated computer-interpretable guideline (CIG) model of a clinical protocol into a temporal hierarchical task network (HTN) planning domain is presented. MATERIALS AND METHODS: The proposed method uses a knowledge-driven reasoning process to translate knowledge previously described in a CIG into a corresponding HTN Planning and Scheduling domain, taking advantage of HTNs known ability to (i) dynamically cope with temporal and resource constraints, and (ii) automatically generate customized plans. The proposed method, focusing on the representation of temporal knowledge and based on the identification of workflow and temporal patterns in a CIG, makes it possible to automatically generate time-annotated and resource-based care pathways tailored to the needs of any possible patient profile. RESULTS: The proposed translation is illustrated through a case study based on a 70 pages long clinical protocol to manage Hodgkin's disease, developed by the Spanish Society of Pediatric Oncology. We show that an HTN planning domain can be generated from the corresponding specification of the protocol in the Asbru language, providing a running example of this translation. Furthermore, the correctness of the translation is checked and also the management of ten different types of temporal patterns represented in the protocol. By interpreting the automatically generated domain with a state-of-art HTN planner, a time-annotated care pathway is automatically obtained, customized for the patient's and institutional needs. The generated care pathway can then be used by clinicians to plan and manage the patients long-term care. CONCLUSION: The described methodology makes it possible to automatically generate patient-tailored care pathways, leveraging an incremental knowledge-driven engineering process that starts from the expert knowledge of medical professionals. The presented approach makes the most of the strengths inherent in both CIG languages and HTN planning and scheduling techniques: for the former, knowledge acquisition and representation of the original clinical protocol, and for the latter, knowledge reasoning capabilities and an ability to deal with complex temporal and resource constraints. Moreover, the proposed approach provides immediate access to technologies such as business process management (BPM) tools, which are increasingly being used to support healthcare processes.

The reproducibility of CLIF, a method for clinical quality indicator formalisation.

In order to be able to automatically calculate clinical quality indicators, we have proposed CLIF, a stepwise method for clinical quality indicator formalisation. Quality indicators are used for external accountability and hospital comparison. As clinical quality indicators are computed in a decentralised manner by the hospitals themselves, reproducibility of the formalisation method is essential to ensure the comparability of calculated values. Thus, we performed a case study to investigate the reproducibility of CLIF. Eight participants formalised the same sample quality indicator with the help of a web-based indicator-authoring tool that facilitates the application of CLIF. We analysed the results per step and concluded that the method itself leads to reproducible results. To further improve reproducibility, ambiguities in the indicator text must be clarified and trained experts are needed to encode clinical concepts and to specify the relations between concepts.

Using model checking for critiquing based on clinical guidelines.

OBJECTIVE: Medical critiquing systems compare clinical actions performed by a physician with a predefined set of actions. In order to provide useful feedback, an important task is to find differences between the actual actions and a set of 'ideal' actions as described by a clinical guideline. In case differences exist, the critiquing system provides insight into the extent to which they are compatible. METHODS AND MATERIAL: We propose a computational method for such critiquing, where the ideal actions are given by a formal model of a clinical guideline, and where the actual actions are derived from real world patient data. We employ model checking to investigate whether a part of the actual treatment is consistent with the guideline. RESULTS: We show how critiquing can be cast in terms of temporal logic, and what can be achieved by using model checking. Furthermore, a method is introduced for off-line computing relevant information which can be exploited during critiquing. The method has been applied to a clinical guideline of breast cancer in conjunction with breast cancer patient data.

Extraction and use of linguistic patterns for modelling medical guidelines.

OBJECTIVE: The quality of knowledge updates in evidence-based medical guidelines can be improved and the effort spent for updating can be reduced if the knowledge underlying the guideline text is explicitly modelled using the so-called linguistic guideline patterns, mappings between a text fragment and a formal representation of its corresponding medical knowledge. METHODS AND MATERIAL: Ontology-driven extraction of linguistic patterns is a method to automatically reconstruct the control knowledge captured in guidelines, which facilitates a more effective modelling and authoring of medical guidelines. We illustrate by examples the use of this method for generating and instantiating linguistic patterns in the text of a guideline for treatment of breast cancer, and evaluate the usefulness of these patterns in the modelling of this guideline. RESULTS: We developed a methodology for extracting and using linguistic patterns in guideline formalization, to aid the human modellers in guideline formalization and reduce the human modelling effort. Using automatic transformation rules for simple linguistic patterns, a good recall (between 72% and 80%) is obtained in selecting the procedural knowledge relevant for the guideline model, even though the precision of the guideline model generated automatically covers only between 20% and 35% of the human-generated guideline model. These results indicate the suitability of our method as a pre-processing step in medical guideline formalization. CONCLUSIONS: Modelling and authoring of medical texts can benefit from our proposed method. As pre-requisites for generating automatically a skeleton of the guideline model from the procedural part of the guideline text, to aid the human modeller, the medical terminology used by the guideline must have a good overlap with existing medical thesauri and its procedural knowledge must obey linguistic regularities that can be mapped into the control constructs of the target guideline modelling language.

Improving medical protocols by formal methods.

OBJECTIVES: During the last decade, evidence-based medicine has given rise to an increasing number of medical practice guidelines and protocols. However, the work done on developing and distributing protocols outweighs the efforts on guaranteeing their quality. Indeed, anomalies like ambiguity and incompleteness are frequent in medical protocols. Recent efforts have tried to address the problem of protocol improvement, but they are not sufficient since they rely on informal processes and notations. Our objective is to improve the quality of medical protocols. APPROACH: The solution we suggest to the problem of quality improvement of protocols consists in the utilisation of formal methods. It requires the definition of an adequate protocol representation language, the development of techniques for the formal analysis of protocols described in that language and, more importantly, the evaluation of the feasibility of the approach based on the formalisation and verification of real-life medical protocols. For the first two aspects we rely on earlier work from the fields of knowledge representation and formal methods. The third aspect, i.e. the evaluation of the use of formal methods in the quality improvement of protocols, constitutes our main objective. The steps with which we have carried out this evaluation are the following: (1) take two real-life reference protocols which cover a wide variety of protocol characteristics; (2) formalise these reference protocols; (3) check the formalisation for the verification of interesting protocol properties; and (4) determine how many errors can be uncovered in this way. RESULTS: Our main results are: a consolidated formal language to model medical practice protocols; two protocols, each both modelled and formalised; a list of properties that medical protocols should satisfy; verification proofs for these protocols and properties; and perspectives of the potentials of this approach. Our results have been evaluated by a panel of medical experts, who judged that the problems we detected in the protocols with the help of formal methods were serious and should be avoided. CONCLUSIONS: We have succeeded in demonstrating the feasibility of formal methods for improving medical protocols.

Protocure: supporting the development of medical protocols through formal methods.

Medical guidelines and protocols describe the optimal care for a specific group of patients and therefore, when properly applied, improve the quality of patient care. During the last decade, a large number of medical guidelines and protocols have been published. However, the work done on developing and disseminating them far outweighs the efforts on guaranteeing their quality. Indeed, anomalies like ambiguity and incompleteness are frequent in medical guidelines and protocols. An approach grounded on a formal representation, can answer these needs, as we have demonstrated in the Protocure project'. The Protocure II project will aim at integrating formal methods in the life cycle of guidelines.