Making Science Computable Using Evidence-Based Medicine on Fast Healthcare Interoperability Resources: Standards Development Project.

BACKGROUND: Evidence-based medicine (EBM) has the potential to improve health outcomes, but EBM has not been widely integrated into the systems used for research or clinical decision-making. There has not been a scalable and reusable computer-readable standard for distributing research results and synthesized evidence among creators, implementers, and the ultimate users of that evidence. Evidence that is more rapidly updated, synthesized, disseminated, and implemented would improve both the delivery of EBM and evidence-based health care policy. OBJECTIVE: This study aimed to introduce the EBM on Fast Healthcare Interoperability Resources (FHIR) project (EBMonFHIR), which is extending the methods and infrastructure of Health Level Seven (HL7) FHIR to provide an interoperability standard for the electronic exchange of health-related scientific knowledge. METHODS: As an ongoing process, the project creates and refines FHIR resources to represent evidence from clinical studies and syntheses of those studies and develops tools to assist with the creation and visualization of FHIR resources. RESULTS: The EBMonFHIR project created FHIR resources (ie, ArtifactAssessment, Citation, Evidence, EvidenceReport, and EvidenceVariable) for representing evidence. The COVID-19 Knowledge Accelerator (COKA) project, now Health Evidence Knowledge Accelerator (HEvKA), took this work further and created FHIR resources that express EvidenceReport, Citation, and ArtifactAssessment concepts. The group is (1) continually refining FHIR resources to support the representation of EBM; (2) developing controlled terminology related to EBM (ie, study design, statistic type, statistical model, and risk of bias); and (3) developing tools to facilitate the visualization and data entry of EBM information into FHIR resources, including human-readable interfaces and JSON viewers. CONCLUSIONS: EBMonFHIR resources in conjunction with other FHIR resources can support relaying EBM components in a manner that is interoperable and consumable by downstream tools and health information technology systems to support the users of evidence.

Representation of evidence-based clinical practice guideline recommendations on FHIR.

BACKGROUND: Various formalisms have been developed to represent clinical practice guideline recommendations in a computer-interpretable way. However, none of the existing formalisms leverage the structured and computable information that emerge from the evidence-based guideline development process. Thus, we here propose a FHIR-based format that uses computer-interpretable representations of the knowledge artifacts that emerge during the process of evidence-based guideline development to directly serve as the basis of evidence-based recommendations. METHODS: We identified the information required to represent evidence-based clinical practice guideline recommendations and reviewed the knowledge artifacts emerging during the evidence-based guideline development process. We then conducted a consensus-based design process with domain experts to develop an information model for guideline recommendation representation that is structurally aligned to the evidence-based guideline recommendation development process and a corresponding representation based on FHIR resources developed for evidence-based medicine (EBMonFHIR). The resulting recommendations were modelled and represented in conformance with the FHIR Clinical Guidelines (CPG-on-FHIR) implementation guide. RESULTS: The information model of evidence-based clinical guideline recommendations and its EBMonFHIR-/CPG-on-FHIR-based representation contain the clinical contents of individual guideline recommendations, a set of metadata for the recommendations, the ratings for the recommendations (e.g., strength of recommendation, certainty of overall evidence), the ratings of certainty of evidence for individual outcomes (e.g., risk of bias) and links to the underlying evidence (systematic reviews based on primary studies). We created profiles and an implementation guide for all FHIR resources required to represent the knowledge artifacts generated during evidence-based guideline development and their re-use as the basis for recommendations and used the profiles to implement an exemplary clinical guideline recommendation. CONCLUSIONS: The FHIR implementation guide presented here can be used to directly link the evidence assessment process of evidence-based guideline recommendation development, i.e. systematic reviews and evidence grading, and the underlying evidence from primary studies to the resulting guideline recommendations. This not only allows the evidence on which recommendations are based on to be evaluated transparently and critically, but also enables guideline developers to leverage computable evidence in a more direct way to facilitate the generation of computer-interpretable guideline recommendations.

A Framework for the Development of Living Practice Guidelines in Health Care.

BACKGROUND: Living practice guidelines are increasingly being used to ensure that recommendations are responsive to rapidly emerging evidence. OBJECTIVE: To develop a framework that characterizes the processes of development of living practice guidelines in health care. DESIGN: First, 3 background reviews were conducted: a scoping review of methods papers, a review of handbooks of guideline-producing organizations, and an analytic review of selected living practice guidelines. Second, the core team drafted the first version of the framework. Finally, the core team refined the framework through an online survey and online discussions with a multidisciplinary international group of stakeholders. SETTING: International. PARTICIPANTS: Multidisciplinary group of 51 persons who have experience with guidelines. MEASUREMENTS: Not applicable. RESULTS: A major principle of the framework is that the unit of update in a living guideline is the individual recommendation. In addition to providing definitions, the framework addresses several processes. The planning process should address the organization's adoption of the living methodology as well as each specific guideline project. The production process consists of initiation, maintenance, and retirement phases. The reporting should cover the evidence surveillance time stamp, the outcome of reassessment of the body of evidence (when applicable), and the outcome of revisiting a recommendation (when applicable). The dissemination process may necessitate the use of different venues, including one for formal publication. LIMITATION: This study does not provide detailed or practical guidance for how the described concepts would be best implemented. CONCLUSION: The framework will help guideline developers in planning, producing, reporting, and disseminating living guideline projects. It will also help research methodologists study the processes of living guidelines. PRIMARY FUNDING SOURCE: None.

Making science computable: Developing code systems for statistics, study design, and risk of bias.

The COVID-19 crisis led a group of scientific and informatics experts to accelerate development of an infrastructure for electronic data exchange for the identification, processing, and reporting of scientific findings. The Fast Healthcare Interoperability Resources (FHIR®) standard which is overcoming the interoperability problems in health information exchange was extended to evidence-based medicine (EBM) knowledge with the EBMonFHIR project. A 13-step Code System Development Protocol was created in September 2020 to support global development of terminologies for exchange of scientific evidence. For Step 1, we assembled expert working groups with 55 people from 26 countries by October 2020. For Step 2, we identified 23 commonly used tools and systems for which the first version of code systems will be developed. For Step 3, a total of 368 non-redundant concepts were drafted to become display terms for four code systems (Statistic Type, Statistic Model, Study Design, Risk of Bias). Steps 4 through 13 will guide ongoing development and maintenance of these terminologies for scientific exchange. When completed, the code systems will facilitate identifying, processing, and reporting research results and the reliability of those results. More efficient and detailed scientific communication will reduce cost and burden and improve health outcomes, quality of life, and patient, caregiver, and healthcare professional satisfaction. We hope the achievements reached thus far will outlive COVID-19 and provide an infrastructure to make science computable for future generations. Anyone may join the effort at https://www.gps.health/covid19_knowledge_accelerator.html.

Processes for evidence summarization for patient decision aids: A Delphi consensus study.

BACKGROUND: Patient decision aids (PDAs) should provide evidence-based information so patients can make informed decisions. Yet, PDA developers do not have an agreed-upon process to select, synthesize and present evidence in PDAs. OBJECTIVE: To reach the consensus on an evidence summarization process for PDAs. DESIGN: A two-round modified Delphi survey. SETTING AND PARTICIPANTS: A group of international experts in PDA development invited developers, scientific networks, patient groups and listservs to complete Delphi surveys. DATA COLLECTION: We emailed participants the study description and a link to the online survey. Participants were asked to rate each potential criterion (omit, possible, desirable, essential) and provide qualitative feedback. ANALYSIS: Criteria in each round were retained if rated by >80% of participants as desirable or essential. If two or more participants suggested rewording, reordering or merging, the steering group considered the suggestion. RESULTS: Following two Delphi survey rounds, the evidence summarization process included defining the decision, reporting the processes and policies of the evidence summarization process, assembling the editorial team and managing (collect, manage, report) their conflicts of interest, conducting a systematic search, selecting and appraising the evidence, presenting the harms and benefits in plain language, and describing the method of seeking external review and the plan for updating the evidence (search, selection and appraisal of new evidence). CONCLUSION: A multidisciplinary stakeholder group reached consensus on an evidence summarization process to guide the creation of high-quality PDAs. PATIENT CONTRIBUTION: A patient partner was part of the steering group and involved in the development of the Delphi survey.

RAPADAPTE for rapid guideline development: high-quality clinical guidelines can be rapidly developed with limited resources.

Guideline development is challenging, expensive and labor-intensive. A high-quality guideline with 90 recommendations for breast cancer treatment was developed within 6 months with limited resources in Costa Rica. We describe the experience and propose a process others can use and adapt.The ADAPTE method (using existing guidelines to minimize repeating work that has been done) was used but existing guidelines were not current. The method was extended to use databases that systematically identify, appraise and synthesize evidence for clinical application (DynaMed, EBM Guidelines) to provide current evidence searches and critical appraisal of evidence. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach was used to rate the quality of evidence and the strength of recommendations. Draft recommendations with supporting evidence were provided to panel members for facilitated voting to target panel discussion to areas necessary for reaching consensus.Training panelists in guideline development methodology facilitated rapid consensus development. Extending 'guideline adaptation' to 'evidence database adaptation' was highly effective and efficient. Methods were created to simplify mapping DynaMed evidence ratings to GRADE ratings. Twelve steps are presented to facilitate rapid guideline development and enable further adaptation by others.This is a case report and the RAPADAPTE method was retrospectively derived. Prospective replication and validation will support advances for the guideline development community. If guideline development can be accelerated without compromising validity and relevance of the resulting recommendations this would greatly improve our ability to impact clinical care.

Longitudinal clinical decision support for assessing decisions over time: State-of-the-art and future directions.

Objective: Patients and clinicians rarely experience healthcare decisions as snapshots in time, but clinical decision support (CDS) systems often represent decisions as snapshots. This scoping review systematically maps challenges and facilitators to longitudinal CDS that are applied at two or more timepoints for the same decision made by the same patient or clinician. Methods: We searched Embase, PubMed, and Medline databases for articles describing development, validation, or implementation of patient- or clinician-facing longitudinal CDS. Validated quality assessment tools were used for article selection. Challenges and facilitators to longitudinal CDS are reported according to PRISMA-ScR guidelines. Results: Eight articles met inclusion criteria; each article described a unique CDS. None used entirely automated data entry, none used living guidelines for updating the evidence base or knowledge engine as new evidence emerged during the longitudinal study, and one included formal readiness for change assessments. Seven of eight CDS were implemented and evaluated prospectively. Challenges were primarily related to suboptimal study design (with unique challenges for each study) or user interface. Facilitators included use of randomized trial designs for prospective enrollment, increased CDS uptake during longitudinal exposure, and machine-learning applications that are tailored to the CDS use case. Conclusions: Despite the intuitive advantages of representing healthcare decisions longitudinally, peer-reviewed literature on longitudinal CDS is sparse. Existing reports suggest opportunities to incorporate longitudinal CDS frameworks, automated data entry, living guidelines, and user readiness assessments. Generating best practice guidelines for longitudinal CDS would require a greater depth and breadth of published work and expert opinion.

Reflections on defining a standard for computable expression of scientific knowledge: What teach us Yoda can.

Science advances at a slow pace but can be accelerated with a standard for computable expression of scientific knowledge, more precisely a technical standard for electronic data exchange of machine-interpretable data expressing scientific knowledge. Efforts to achieve this vision include Evidence-Based Medicine on Fast Healthcare Interoperability Resources (EBMonFHIR), COVID-19 Knowledge Accelerator (COKA), Computable Publishing LLC, Scientific Knowledge Accelerator Foundation, and the Fast Evidence Interoperability Resources (FEvIR) Platform. The vision for communicating scientific research results to be instantly found, viewed, sent, received, and incorporated into thousands of system is a Just-in-time Evidence Dissemination and Integration (JEDI) vision. Reflections on JEDI teachings in a fun way helps explain the focus on the "Resource" to make science computable, how a precise specification of evidence changes perspectives, the need to reset standard terminologies, the inclusive nature of efforts to achieve the vision, and the critical demand to realize the vision.

Striking a match between FHIR-based patient data and FHIR-based eligibility criteria.

Inputs and Outputs: The Strike-a-Match Function, written in JavaScript version ES6+, accepts the input of two datasets (one dataset defining eligibility criteria for research studies or clinical decision support, and one dataset defining characteristics for an individual patient). It returns an output signaling whether the patient characteristics are a match for the eligibility criteria. Purpose: Ultimately, such a system will play a "matchmaker" role in facilitating point-of-care recognition of patient-specific clinical decision support. Specifications: The eligibility criteria are defined in HL7 FHIR (version R5) EvidenceVariable Resource JSON structure. The patient characteristics are provided in an FHIR Bundle Resource JSON including one Patient Resource and one or more Observation and Condition Resources which could be obtained from the patient's electronic health record. Application: The Strike-a-Match Function determines whether or not the patient is a match to the eligibility criteria and an Eligibility Criteria Matching Software Demonstration interface provides a human-readable display of matching results by criteria for the clinician or patient to consider. This is the first software application, serving as proof of principle, that compares patient characteristics and eligibility criteria with all data exchanged using HL7 FHIR JSON. An Eligibility Criteria Matching Software Library at https://fevir.net/110192 provides a method for sharing functions using the same information model.

Clinical Quality Measures: A Challenge for-and to-Family Physicians.

INTRODUCTION: Improving design, selection and implementation of appropriate clinical quality measures can reduce harms and costs of health care and improve the quality and experience of care delivery. These measures have not been evaluated for appropriateness for use in performance measurement in a systematic, reproducible, and widely accepted manner. METHODS: We defined 10 criteria for evaluation of measure appropriateness in 4 domains: Patient-centeredness of outcomes, specification of population measured and measure detail, reliable evidence that benefits likely outweigh harms, and independence from significant confounders. We applied these criteria to 24 measures under consideration for statewide use in Massachusetts in public and private incentive-based programs. We appraised each measure as Appropriate or Not Appropriate for such use. RESULTS: We rated 15 measures as Appropriate (62.5%). Three measures (12.5%) were considered Appropriate only if applied at a system level but not for patient-provider assessment and 6 measures (25%) were rated Not Appropriate. Reasons for designation as "Not Appropriate" included benefits not clearly outweighing harms, lack of preservation of patient autonomy, inappropriate specification of population and measure detail, confounding by locus of control, and confounding by social determinants of health. CONCLUSIONS: Using this consensus-driven, 10-criteria methodology we were able to evaluate appropriateness of clinical quality measures. This methodology may improve measure design and inform selection of the most appropriate measures for use in quality measurement, financial incentives, and reporting.

Categorizing metadata to help mobilize computable biomedical knowledge.

INTRODUCTION: Computable biomedical knowledge artifacts (CBKs) are digital objects conveying biomedical knowledge in machine-interpretable structures. As more CBKs are produced and their complexity increases, the value obtained from sharing CBKs grows. Mobilizing CBKs and sharing them widely can only be achieved if the CBKs are findable, accessible, interoperable, reusable, and trustable (FAIR+T). To help mobilize CBKs, we describe our efforts to outline metadata categories to make CBKs FAIR+T. METHODS: We examined the literature regarding metadata with the potential to make digital artifacts FAIR+T. We also examined metadata available online today for actual CBKs of 12 different types. With iterative refinement, we came to a consensus on key categories of metadata that, when taken together, can make CBKs FAIR+T. We use subject-predicate-object triples to more clearly differentiate metadata categories. RESULTS: We defined 13 categories of CBK metadata most relevant to making CBKs FAIR+T. Eleven of these categories (type, domain, purpose, identification, location, CBK-to-CBK relationships, technical, authorization and rights management, provenance, evidential basis, and evidence from use metadata) are evident today where CBKs are stored online. Two additional categories (preservation and integrity metadata) were not evident in our examples. We provide a research agenda to guide further study and development of these and other metadata categories. CONCLUSION: A wide variety of metadata elements in various categories is needed to make CBKs FAIR+T. More work is needed to develop a common framework for CBK metadata that can make CBKs FAIR+T for all stakeholders.

Developing trustworthy recommendations as part of an urgent response (1-2 weeks): a GRADE concept paper.

OBJECTIVES: The aim of this study is to propose an approach for developing trustworthy recommendations as part of urgent responses (1-2 week) in the clinical, public health, and health systems fields. STUDY DESIGN AND SETTING: We conducted a review of the literature, outlined a draft approach, refined the concept through iterative discussions, a workshop by the Grading of Recommendations Assessment, Development and Evaluation Rapid Guidelines project group, and obtained feedback from the larger Grading of Recommendations Assessment, Development and Evaluation working group. RESULTS: A request for developing recommendations within 2 week is the usual trigger for an urgent response. Although the approach builds on the general principles of trustworthy guideline development, we highlight the following steps: (1) assess the level of urgency; (2) assess feasibility; (3) set up the organizational logistics; (4) specify the question(s); (5) collect the information needed; (6) assess the adequacy of identified information; (7) develop the recommendations using one of the 4 potential approaches: adopt existing recommendations, adapt existing recommendations, develop new recommendations using existing adequate systematic review, or develop new recommendations using expert panel input; and (8) consider an updating plan. CONCLUSION: An urgent response for developing recommendations requires building a cohesive, skilled, and highly motivated multidisciplinary team with the necessary clinical, scientific, and methodological expertise; adapting to shifting needs; complying with the principles of transparency; and properly managing conflicts of interest.

Review of online evidence-based practice point-of-care information summary providers: response by the publisher of DynaMed.

In response to Banzi's et al review of online evidence-based practice point-of-care resources published in the Journal of Medical Internet Research, the publisher of DynaMed clarifies his evidence-based methodology.

Modified Delphi survey for the evidence summarisation of patient decision aids: Study protocol.

INTRODUCTION: Information included in a patient decision aid (PDA) can significantly influence patients' decisions and is, therefore, expected to be evidence-based and rigorously selected and summarised. PDA developers have not yet agreed on a standardised process for the selection and summarisation of the supporting evidence. We intend to generate consensus on a process (and related steps and criteria) for selecting and summarising evidence for PDAs using a modified Delphi survey. METHODS AND ANALYSIS: We will develop an evidence summarisation process specific to PDA development by using a consensus-based Delphi approach, surveying international experts and stakeholders with two to three rounds. To increase generalisability and acceptability, we will distribute the survey to the following stakeholder groups: PDA developers, researchers with expertise in shared decision making, PDA development and evidence summarisation, members of the International Patient Decision Aids Standards (IPDAS) collaboration, policy makers with expertise in PDA certification and patient stakeholder groups. For each criterion, if at least 80% of survey participants rank the criterion as most important/least important, we will consider that consensus has been achieved. ETHICS AND DISSEMINATION: It is critical for PDAs to have accurate and trustworthy evidence-based information about the risks and benefits of health treatments and tests, as these decision aids help patients make important choices. We want to generate consensus on an approach for selecting and summarising the evidence included in PDAs, which can be widely implemented by PDA developers. Dartmouth College's Committee for the Protection of Human Subjects approved this protocol. We will publish our results in a peer reviewed journal.