Beyond mathematics, statistics, and programming: data science, machine learning, and artificial intelligence competencies and curricula for clinicians, informaticians, science journalists, and researchers.

Data science, machine learning and artificial intelligence applications impact clinicians, informaticians, science journalists, and researchers. Most biomedical data science training focuses on learning a programming language in addition to higher mathematics and advanced statistics. This approach is appropriate for graduate students but greatly reduces the number of individuals in healthcare who can be involved in data science. To serve these four stakeholder audiences, we describe several curricular strategies focusing on solving real problems of interest to these audiences. Relevant competencies for these audiences include using intuitive programming tools that facilitate data exploration with minimal programming background, creating data models, evaluating results of data analyses, and assessing data science research reports, among others. Offering the curricula described here more broadly could broaden the stakeholder groups knowledgeable about and engaged in data science.

The IMPACT framework and implementation for accessible in silico clinical phenotyping in the digital era.

Clinical phenotyping is often a foundational requirement for obtaining datasets necessary for the development of digital health applications. Traditionally done via manual abstraction, this task is often a bottleneck in development due to time and cost requirements, therefore raising significant interest in accomplishing this task via in-silico means. Nevertheless, current in-silico phenotyping development tends to be focused on a single phenotyping task resulting in a dearth of reusable tools supporting cross-task generalizable in-silico phenotyping. In addition, in-silico phenotyping remains largely inaccessible for a substantial portion of potentially interested users. Here, we highlight the barriers to the usage of in-silico phenotyping and potential solutions in the form of a framework of several desiderata as observed during our implementation of such tasks. In addition, we introduce an example implementation of said framework as a software application, with a focus on ease of adoption, cross-task reusability, and facilitating the clinical phenotyping algorithm development process.

Clinical study applying machine learning to detect a rare disease: results and lessons learned.

Machine learning has the potential to improve identification of patients for appropriate diagnostic testing and treatment, including those who have rare diseases for which effective treatments are available, such as acute hepatic porphyria (AHP). We trained a machine learning model on 205 571 complete electronic health records from a single medical center based on 30 known cases to identify 22 patients with classic symptoms of AHP that had neither been diagnosed nor tested for AHP. We offered urine porphobilinogen testing to these patients via their clinicians. Of the 7 who agreed to testing, none were positive for AHP. We explore the reasons for this and provide lessons learned for further work evaluating machine learning to detect AHP and other rare diseases.

The International Academy of Health Sciences Informatics: 2021 Update.

OBJECTIVES: To summarize the activities of the International Academy of Health Sciences Informatics (IAHSI) in 2021 and welcome its 2021 Class of Fellows. METHODS: Report on governance, strategic directions, newly elected fellows, plenary meetings, and other activities of the Academy. RESULTS: As in 2020, all of the Academy's activities were carried out virtually due to the COVID-19 pandemic. In 2021, new Board members were elected. Strategic activities in data standards and interoperability and in mentorship moved forward. A new class of 26 Fellows was elected, bringing the total membership of the Academy to 204 Fellows from all regions of the world. In addition, a virtual plenary meeting was held. CONCLUSIONS: The Academy has continued to pursue its role as the honorific society globally for biomedical and health informatics. Expansion of strategic activities and membership will continue moving forward.

Searching for scientific evidence in a pandemic: An overview of TREC-COVID.

We present an overview of the TREC-COVID Challenge, an information retrieval (IR) shared task to evaluate search on scientific literature related to COVID-19. The goals of TREC-COVID include the construction of a pandemic search test collection and the evaluation of IR methods for COVID-19. The challenge was conducted over five rounds from April to July 2020, with participation from 92 unique teams and 556 individual submissions. A total of 50 topics (sets of related queries) were used in the evaluation, starting at 30 topics for Round 1 and adding 5 new topics per round to target emerging topics at that state of the still-emerging pandemic. This paper provides a comprehensive overview of the structure and results of TREC-COVID. Specifically, the paper provides details on the background, task structure, topic structure, corpus, participation, pooling, assessment, judgments, results, top-performing systems, lessons learned, and benchmark datasets.

The International Academy of Health Sciences Informatics (IAHSI): 2020 Report.

OBJECTIVES: To summarize the major activities of the International Academy of Health Sciences Informatics (IAHSI) in the 2020 time period and to welcome its 2020 Class of Fellows. METHOD: Report from the members of the Academy's Board. RESULTS: Due to the SARS-CoV-2 pandemic, both Plenary meetings in 2020 had to be organized as virtual meetings. Scientific discussions, focusing on mobilizing computable biomedical knowledge and on data standards and interoperability formed major parts of these meetings. A statement on the use of informatics in pandemic situations was elaborated and sent to the World Health Organization. A panel on data standards and interoperability started its work. 34 Fellows were welcomed in the 2020 Class of Fellows so that the Academy now consists of 179 members. CONCLUSIONS: There was a shift from supporting to strategic activities in the Academy's work. After having achieved organizational stability, the Academy can now focus on its strategic work and so on its main objective.

Digital Health during COVID-19: Informatics Dialogue with the World Health Organization.

BACKGROUND: On December 16, 2020 representatives of the International Medical Informatics Association (IMIA), a Non-Governmental Organization in official relations with the World Health Organization (WHO), along with its International Academy for Health Sciences Informatics (IAHSI), held an open dialogue with WHO Director General (WHO DG) Tedros Adhanom Ghebreyesus about the opportunities and challenges of digital health during the COVID-19 global pandemic. OBJECTIVES: The aim of this paper is to report the outcomes of the dialogue and discussions with more than 200 participants representing different civil society organizations (CSOs). METHODS: The dialogue was held in form of a webinar. After an initial address of the WHO DG, short presentations by the panelists, and live discussions between panelists, the WHO DG and WHO representatives took place. The audience was able to post questions in written. These written discussions were saved with participants' consent and summarized in this paper. RESULTS: The main themes that were brought up by the audience for discussion were: (a) opportunities and challenges in general; (b) ethics and artificial intelligence; (c) digital divide; (d) education. Proposed actions included the development of a roadmap based on the lessons learned from the COVID-19 pandemic. CONCLUSIONS: Decision making by policy makers needs to be evidence-based and health informatics research should be used to support decisions surrounding digital health, and we further propose next steps in the collaboration between IMIA and WHO such as future engagement in the World Health Assembly.

A comparative analysis of system features used in the TREC-COVID information retrieval challenge.

The COVID-19 pandemic has resulted in a rapidly growing quantity of scientific publications from journal articles, preprints, and other sources. The TREC-COVID Challenge was created to evaluate information retrieval (IR) methods and systems for this quickly expanding corpus. Using the COVID-19 Open Research Dataset (CORD-19), several dozen research teams participated in over 5 rounds of the TREC-COVID Challenge. While previous work has compared IR techniques used on other test collections, there are no studies that have analyzed the methods used by participants in the TREC-COVID Challenge. We manually reviewed team run reports from Rounds 2 and 5, extracted features from the documented methodologies, and used a univariate and multivariate regression-based analysis to identify features associated with higher retrieval performance. We observed that fine-tuning datasets with relevance judgments, MS-MARCO, and CORD-19 document vectors was associated with improved performance in Round 2 but not in Round 5. Though the relatively decreased heterogeneity of runs in Round 5 may explain the lack of significance in that round, fine-tuning has been found to improve search performance in previous challenge evaluations by improving a system's ability to map relevant queries and phrases to documents. Furthermore, term expansion was associated with improvement in system performance, and the use of the narrative field in the TREC-COVID topics was associated with decreased system performance in both rounds. These findings emphasize the need for clear queries in search. While our study has some limitations in its generalizability and scope of techniques analyzed, we identified some IR techniques that may be useful in building search systems for COVID-19 using the TREC-COVID test collections.

Evaluation of patient-level retrieval from electronic health record data for a cohort discovery task.

OBJECTIVE: Growing numbers of academic medical centers offer patient cohort discovery tools to their researchers, yet the performance of systems for this use case is not well understood. The objective of this research was to assess patient-level information retrieval methods using electronic health records for different types of cohort definition retrieval. MATERIALS AND METHODS: We developed a test collection consisting of about 100 000 patient records and 56 test topics that characterized patient cohort requests for various clinical studies. Automated information retrieval tasks using word-based approaches were performed, varying 4 different parameters for a total of 48 permutations, with performance measured using B-Pref. We subsequently created structured Boolean queries for the 56 topics for performance comparisons. In addition, we performed a more detailed analysis of 10 topics. RESULTS: The best-performing word-based automated query parameter settings achieved a mean B-Pref of 0.167 across all 56 topics. The way a topic was structured (topic representation) had the largest impact on performance. Performance not only varied widely across topics, but there was also a large variance in sensitivity to parameter settings across the topics. Structured queries generally performed better than automated queries on measures of recall and precision but were still not able to recall all relevant patients found by the automated queries. CONCLUSION: While word-based automated methods of cohort retrieval offer an attractive solution to the labor-intensive nature of this task currently used at many medical centers, we generally found suboptimal performance in those approaches, with better performance obtained from structured Boolean queries. Future work will focus on using the test collection to develop and evaluate new approaches to query structure, weighting algorithms, and application of semantic methods.

Addressing the Search Challenges of Precision Medicine with Information Retrieval Systems and Physician Readers.

The Text REtrieval Conference (TREC), co-sponsored by the National Institute of Standards and Technology (NIST) in the US and US Department of Defense, was started in 1992. TREC's purpose is to support research within the information retrieval community by providing the infrastructure necessary for large-scale evaluation of text retrieval methodologies. In 2017, the TREC Precision Medicine (Roberts et al., 2017) track grew from the Clinical Decision Support track and focused on a narrower problem domain of precision oncology. After three years of computer runs being evaluated for relevance by physician readers, we provide a unique perspective of how to evaluate computer-generated articles and clinical trials pulled from PubMed and Clinicaltrials.gov to find relevant information on medical cases.

TREC-COVID: rationale and structure of an information retrieval shared task for COVID-19.

TREC-COVID is an information retrieval (IR) shared task initiated to support clinicians and clinical research during the COVID-19 pandemic. IR for pandemics breaks many normal assumptions, which can be seen by examining 9 important basic IR research questions related to pandemic situations. TREC-COVID differs from traditional IR shared task evaluations with special considerations for the expected users, IR modality considerations, topic development, participant requirements, assessment process, relevance criteria, evaluation metrics, iteration process, projected timeline, and the implications of data use as a post-task test collection. This article describes how all these were addressed for the particular requirements of developing IR systems under a pandemic situation. Finally, initial participation numbers are also provided, which demonstrate the tremendous interest the IR community has in this effort.

Protecting healing relationships in the age of electronic health records: report from an international conference.

We present findings of an international conference of diverse participants exploring the influence of electronic health records (EHRs) on the patient-practitioner relationship. Attendees united around a belief in the primacy of this relationship and the importance of undistracted attention. They explored administrative, regulatory, and financial requirements that have guided United States (US) EHR design and challenged patient-care documentation, usability, user satisfaction, interconnectivity, and data sharing. The United States experience was contrasted with those of other nations, many of which have prioritized patient-care documentation rather than billing requirements and experienced high user satisfaction. Conference participants examined educational methods to teach diverse learners effective patient-centered EHR use, including alternative models of care delivery and documentation, and explored novel ways to involve patients as healthcare partners like health-data uploading, chart co-creation, shared practitioner notes, applications, and telehealth. Future best practices must preserve human relationships, while building an effective patient-practitioner (or team)-EHR triad.

Five Years of Clinical Informatics Board Certification for Physicians in the United States of America.

OBJECTIVES: To review the highlights of the new Clinical Informatics subspecialty including its history, certification requirements, development of and performance on the certification examination in the United States. METHODS: We reviewed processes for the development of a subspecialty. Data from board certification examinations were collated and analyzed. We discussed eligibility requirements in the fellowship as well as practice pathways. RESULTS: Lessons learned from the development of the Clinical Informatics subspecialty, opportunities, challenges, and future directions for the field are discussed. CONCLUSIONS: There remains a need for fellowship programs and creation and maintenance of a professional home for the subspecialty with the American Medical Informatics Association. Ongoing attention to the currency of the core content is required to maintain an examination designed to test the key concepts within the field of Clinical Informatics.

Characteristics of the healthcare information technology workforce in the HITECH era: underestimated in size, still growing, and adapting to advanced uses.

OBJECTIVE: There is little readily available data about the size and characteristics of the healthcare information technology workforce. We sought to update a previous description of the size, growth, and characteristics of this workforce based on the Healthcare Information Management Systems Society (HIMSS) Analytics® Database, a resource that includes hospital size, number of beds, amount of staffing, and an eight-stage model of electronic health record adoption (Electronic Medical Record Adoption Model, EMRAM℠). MATERIALS AND METHODS: We updated an analysis done using a 2007 snapshot of the HIMSS Analytics Database with a comparable snapshot from 2014 in order to estimate the size of the current workforce and project future needs. For the 2014 data, we applied the same weighted average analysis used in 2007 to obtain a ratio of information technology (IT) hospital full-time equivalent (FTE) to staffed beds, extrapolate the results to all US hospitals, and project the workforce needs as hospitals achieve higher EMRAM stages. RESULTS: Our estimated size of the healthcare information technology workforce in the US in 2014 was 161 160, which was 8.0% larger than the estimate based on the 2007 data. Based on the new data, we project a potential need for an additional 19 852 and 153 114 FTE, if all hospitals were to achieve EMRAM Stages 6 and 7, respectively. The distribution of FTE across job function category varies by EMRAM stage. DISCUSSION AND CONCLUSIONS: Although these data are limited, especially for EMRAM Stage 7, there is likely need for substantial workforce growth as hospitals increase their adoption of advanced healthcare information technology. Further research with data better focused on workforce characteristics will provide a better picture of staffing requirements.

A publicly available benchmark for biomedical dataset retrieval: the reference standard for the 2016 bioCADDIE dataset retrieval challenge.

Database URL: https://biocaddie.org/benchmark-data.

Health Information Exchange Use (1990-2015): A Systematic Review.

BACKGROUND: In June 2014, the Office of the National Coordinator for Health Information Technology published a 10-year roadmap for the United States to achieve interoperability of electronic health records (EHR) by 2024. A key component of this strategy is the promotion of nationwide health information exchange (HIE). The 2009 Health Information Technology for Economic and Clinical Health (HITECH) Act provided significant investments to achieve HIE. OBJECTIVE: We conducted a systematic literature review to describe the use of HIE through 2015. METHODS: We searched MEDLINE, PsycINFO, CINAHL, and Cochrane databases (1990 - 2015); reference lists; and tables of contents of journals not indexed in the databases searched. We extracted data describing study design, setting, geographic location, characteristics of HIE implementation, analysis, follow-up, and results. Study quality was dual-rated using pre-specified criteria and discrepancies resolved through consensus. RESULTS: We identified 58 studies describing either level of use or primary uses of HIE. These were a mix of surveys, retrospective database analyses, descriptions of audit logs, and focus groups. Settings ranged from community-wide to multinational. Results suggest that HIE use has risen substantially over time, with 82% of non-federal hospitals exchanging information (2015), 38% of physician practices (2013), and 17-23% of long-term care facilities (2013). Statewide efforts, originally funded by HITECH, varied widely, with a small number of states providing the bulk of the data. Characteristics of greater use include the presence of an EHR, larger practice size, and larger market share of the health-system. CONCLUSIONS: Use of HIE in the United States is growing but is still limited. Opportunities remain for expansion. Characteristics of successful implementations may provide a path forward.