Recommendations to overcome barriers to the use of artificial intelligence-driven evidence in health technology assessment.

Background: Artificial intelligence (AI) has attracted much attention because of its enormous potential in healthcare, but uptake has been slow. There are substantial barriers that challenge health technology assessment (HTA) professionals to use AI-generated evidence for decision-making from large real-world databases (e.g., based on claims data). As part of the European Commission-funded HTx H2020 (Next Generation Health Technology Assessment) project, we aimed to put forward recommendations to support healthcare decision-makers in integrating AI into the HTA processes. The barriers, addressed by the paper, are particularly focusing on Central and Eastern European (CEE) countries, where the implementation of HTA and access to health databases lag behind Western European countries. Methods: We constructed a survey to rank the barriers to using AI for HTA purposes, completed by respondents from CEE jurisdictions with expertise in HTA. Using the results, two members of the HTx consortium from CEE developed recommendations on the most critical barriers. Then these recommendations were discussed in a workshop by a wider group of experts, including HTA and reimbursement decision-makers from both CEE countries and Western European countries, and summarized in a consensus report. Results: Recommendations have been developed to address the top 15 barriers in areas of (1) human factor-related barriers, focusing on educating HTA doers and users, establishing collaborations and best practice sharing; (2) regulatory and policy-related barriers, proposing increasing awareness and political commitment and improving the management of sensitive information for AI use; (3) data-related barriers, suggesting enhancing standardization and collaboration with data networks, managing missing and unstructured data, using analytical and statistical approaches to address bias, using quality assessment tools and quality standards, improving reporting, and developing better conditions for the use of data; and (4) technological barriers, suggesting sustainable development of AI infrastructure. Conclusion: In the field of HTA, the great potential of AI to support evidence generation and evaluation has not yet been sufficiently explored and realized. Raising awareness of the intended and unintended consequences of AI-based methods and encouraging political commitment from policymakers is necessary to upgrade the regulatory and infrastructural environment and knowledge base required to integrate AI into HTA-based decision-making processes better.

Guidance on using real-world evidence from Western Europe in Central and Eastern European health policy decision making.

Aim: Real-world data and real-world evidence (RWE) are becoming more important for healthcare decision making and health technology assessment. We aimed to propose solutions to overcome barriers preventing Central and Eastern European (CEE) countries from using RWE generated in Western Europe. Materials & methods: To achieve this, following a scoping review and a webinar, the most important barriers were selected through a survey. A workshop was held with CEE experts to discuss proposed solutions. Results: Based on survey results, we selected the nine most important barriers. Multiple solutions were proposed, for example, the need for a European consensus, and building trust in using RWE. Conclusion: Through collaboration with regional stakeholders, we proposed a list of solutions to overcome barriers on transferring RWE from Western Europe to CEE countries.

Investigating the Scientific 'Infodemic' Phenomenon Related to the COVID-19 Pandemic.

OBJECTIVES: The study aims at understanding the structural characteristics and content features of COVID-19 literature and public health data from the perspective of the 'Language and Meaning in Biomedicine' Working Group (LaMB WG) of IMIA. The LaMB WG has interest in conceptual characteristics, transparency, comparability, and reusability of medical information, both in science and practice. METHODS: A set of methods were used (i) investigating the overall speed and dynamics of COVID-19 publications; (ii) characterizing the concepts of COVID-19 (text mining, visualizing a semantic map of related concepts); (iii) assessing (re)usability and combinability of data sets and paper collections (as textual data sets), and checking if information is Findable, Accessible, Interoperable, and Reusable (FAIR). A further method tested practical usability of FAIR requirements by setting up a common data space of epidemiological, virus genetics and governmental public health measures' stringency data of various origin, where complex data points were visualized as scatter plots. RESULTS: Never before were that many papers and data sources dedicated to one pandemic. Worldwide research shows a plateau at ∼ 2,200 papers per week - the dynamics of areas of studies being slightly different. Ratio of epidemic modelling is rather low (∼1%). A few 'language and meaning' methods, such as using integrated terminologies, applying data and metadata standards for processing epidemiological and case-related clinical information and in general, principles of FAIR data handling could contribute to better results, such as improved interoperability and meaningful knowledge sharing in a virtuous cycle of continuous improvements.

The Interplay of Knowledge Representation with Various Fields of Artificial Intelligence in Medicine.

INTRODUCTION: Artificial intelligence (AI) is widespread in many areas, including medicine. However, it is unclear what exactly AI encompasses. This paper aims to provide an improved understanding of medical AI and its constituent fields, and their interplay with knowledge representation (KR). METHODS: We followed a Wittgensteinian approach ("meaning by usage") applied to content metadata labels, using the Medical Subject Headings (MeSH) thesaurus to classify the field. To understand and characterize medical AI and the role of KR, we analyzed: (1) the proportion of papers in MEDLINE related to KR and various AI fields; (2) the interplay among KR and AI fields and overlaps among the AI fields; (3) interconnectedness of fields; and (4) phrase frequency and collocation based on a corpus of abstracts. RESULTS: Data from over eighty thousand papers showed a steep, six-fold surge in the last 30 years. This growth happened in an escalating and cascading way. A corpus of 246,308 total words containing 21,842 unique words showed several hundred occurrences of notions such as robotics, fuzzy logic, neural networks, machine learning and expert systems in the phrase frequency analysis. Collocation analysis shows that fuzzy logic seems to be the most often collocated notion. Neural networks and machine learning are also used in the conceptual neighborhood of KR. Robotics is more isolated. CONCLUSIONS: Authors note an escalation of published AI studies in medicine. Knowledge representation is one of the smaller areas, but also the most interconnected, and provides a common cognitive layer for other areas.

Determinants and Drivers of Infectious Disease Threat Events in Europe.

Infectious disease threat events (IDTEs) are increasing in frequency worldwide. We analyzed underlying drivers of 116 IDTEs detected in Europe during 2008-2013 by epidemic intelligence at the European Centre of Disease Prevention and Control. Seventeen drivers were identified and categorized into 3 groups: globalization and environment, sociodemographic, and public health systems. A combination of >2 drivers was responsible for most IDTEs. The driver category globalization and environment contributed to 61% of individual IDTEs, and the top 5 individual drivers of all IDTEs were travel and tourism, food and water quality, natural environment, global trade, and climate. Hierarchical cluster analysis of all drivers identified travel and tourism as a distinctly separate driver. Monitoring and modeling such disease drivers can help anticipate future IDTEs and strengthen control measures. More important, intervening directly on these underlying drivers can diminish the likelihood of the occurrence of an IDTE and reduce the associated human and economic costs.

Using the EC decision on case definitions for communicable diseases as a terminology source--lessons learned.

Extracting scientifically accurate terminology from an EU public health regulation is part of the knowledge engineering work at the European Centre for Disease Prevention and Control (ECDC). ECDC operates information systems at the crossroads of many areas - posing a challenge for transparency and consistency. Semantic interoperability is based on the Terminology Server (TS). TS value sets (structured vocabularies) describe shared domains as "diseases", "organisms", "public health terms", "geo-entities" "organizations" and "administrative terms" and others. We extracted information from the relevant EC Implementing Decision on case definitions for reporting communicable diseases, listing 53 notifiable infectious diseases, containing clinical, diagnostic, laboratory and epidemiological criteria. We performed a consistency check; a simplification - abstraction; we represented lab criteria in triplets: as 'y' procedural result /of 'x' organism-substance/on 'z' specimen and identified negations. The resulting new case definition value set represents the various formalized criteria, meanwhile the existing disease value set has been extended, new signs and symptoms were added. New organisms enriched the organism value set. Other new categories have been added to the public health value set, as transmission modes; substances; specimens and procedures. We identified problem areas, as (a) some classification error(s); (b) inconsistent granularity of conditions; (c) seemingly nonsense criteria, medical trivialities; (d) possible logical errors, (e) seemingly factual errors that might be phrasing errors. We think our hypothesis regarding room for possible improvements is valid: there are some open issues and a further improved legal text might lead to more precise epidemiologic data collection. It has to be noted that formal representation for automatic classification of cases was out of scope, such a task would require other formalism, as e.g. those used by rule-based decision support systems.

Medical concept representation: the years beyond 2000.

This work aims at understanding the state of the art in the broad contextual research area of "medical concept representation". Our data support the general understanding that the focus of research has moved toward medical ontologies, which we interpret as a paradigm shift. Both the opinion of socially active groups of researchers and changes in bibliometric data since 1988 support this opinion. Socially active researchers mention the OBO foundry, SNOMED CT, and the UMLS as anchor activities.

From concept representations to ontologies: a paradigm shift in health informatics?

OBJECTIVES: This work aims at uncovering challenges in biomedical knowledge representation research by providing an understanding of what was historically called "medical concept representation" and used as the name for a working group of the International Medical Informatics Association. METHODS: Bibliometrics, text mining, and a social media survey compare the research done in this area between two periods, before and after 2000. RESULTS: Both the opinion of socially active groups of researchers and the interpretation of bibliometric data since 1988 suggest that the focus of research has moved from "medical concept representation" to "medical ontologies". CONCLUSIONS: It remains debatable whether the observed change amounts to a paradigm shift or whether it simply reflects changes in naming, following the natural evolution of ontology research and engineering activities in the 1990s. The availability of powerful tools to handle ontologies devoted to certain areas of biomedicine has not resulted in a large-scale breakthrough beyond advances in basic research.

The 'cube' meta-model for the information system of large health sector organizations--a (platform neutral) mapping tool to integrate information system development with changing business functions and organizational development.

To develop information systems (IS) in the changing environment of the health sector, a simple but throughout model, avoiding the techno-jargon of informatics, might be useful for the top management. A platform neutral, extensible, transparent conceptual model should be established. Limitations of current methods lead to a simple, but comprehensive mapping, in the form of a three-dimensional cube. The three 'orthogonal' views are (a) organization functionality, (b) organizational structures and (c) information technology. Each of the cube-sides is described according to its nature. This approach enables to define any kind of an IS component as a certain point/layer/domain of the cube and enables also the management to label all IS components independently form any supplier(s) and/or any specific platform. The model handles changes in organization structure, business functionality and the serving info-system independently form each other. Practical application extends to (a) planning complex, new ISs, (b) guiding development of multi-vendor, multi-site ISs, (c) supporting large-scale public procurement procedures and the contracting, implementation phase by establishing a platform neutral reference, (d) keeping an exhaustive inventory of an existing large-scale system, that handles non-tangible aspects of the IS.