Making Science Computable Using Evidence-Based Medicine on Fast Healthcare Interoperability Resources: Standards Development Project.

BACKGROUND: Evidence-based medicine (EBM) has the potential to improve health outcomes, but EBM has not been widely integrated into the systems used for research or clinical decision-making. There has not been a scalable and reusable computer-readable standard for distributing research results and synthesized evidence among creators, implementers, and the ultimate users of that evidence. Evidence that is more rapidly updated, synthesized, disseminated, and implemented would improve both the delivery of EBM and evidence-based health care policy. OBJECTIVE: This study aimed to introduce the EBM on Fast Healthcare Interoperability Resources (FHIR) project (EBMonFHIR), which is extending the methods and infrastructure of Health Level Seven (HL7) FHIR to provide an interoperability standard for the electronic exchange of health-related scientific knowledge. METHODS: As an ongoing process, the project creates and refines FHIR resources to represent evidence from clinical studies and syntheses of those studies and develops tools to assist with the creation and visualization of FHIR resources. RESULTS: The EBMonFHIR project created FHIR resources (ie, ArtifactAssessment, Citation, Evidence, EvidenceReport, and EvidenceVariable) for representing evidence. The COVID-19 Knowledge Accelerator (COKA) project, now Health Evidence Knowledge Accelerator (HEvKA), took this work further and created FHIR resources that express EvidenceReport, Citation, and ArtifactAssessment concepts. The group is (1) continually refining FHIR resources to support the representation of EBM; (2) developing controlled terminology related to EBM (ie, study design, statistic type, statistical model, and risk of bias); and (3) developing tools to facilitate the visualization and data entry of EBM information into FHIR resources, including human-readable interfaces and JSON viewers. CONCLUSIONS: EBMonFHIR resources in conjunction with other FHIR resources can support relaying EBM components in a manner that is interoperable and consumable by downstream tools and health information technology systems to support the users of evidence.

Representation of evidence-based clinical practice guideline recommendations on FHIR.

BACKGROUND: Various formalisms have been developed to represent clinical practice guideline recommendations in a computer-interpretable way. However, none of the existing formalisms leverage the structured and computable information that emerge from the evidence-based guideline development process. Thus, we here propose a FHIR-based format that uses computer-interpretable representations of the knowledge artifacts that emerge during the process of evidence-based guideline development to directly serve as the basis of evidence-based recommendations. METHODS: We identified the information required to represent evidence-based clinical practice guideline recommendations and reviewed the knowledge artifacts emerging during the evidence-based guideline development process. We then conducted a consensus-based design process with domain experts to develop an information model for guideline recommendation representation that is structurally aligned to the evidence-based guideline recommendation development process and a corresponding representation based on FHIR resources developed for evidence-based medicine (EBMonFHIR). The resulting recommendations were modelled and represented in conformance with the FHIR Clinical Guidelines (CPG-on-FHIR) implementation guide. RESULTS: The information model of evidence-based clinical guideline recommendations and its EBMonFHIR-/CPG-on-FHIR-based representation contain the clinical contents of individual guideline recommendations, a set of metadata for the recommendations, the ratings for the recommendations (e.g., strength of recommendation, certainty of overall evidence), the ratings of certainty of evidence for individual outcomes (e.g., risk of bias) and links to the underlying evidence (systematic reviews based on primary studies). We created profiles and an implementation guide for all FHIR resources required to represent the knowledge artifacts generated during evidence-based guideline development and their re-use as the basis for recommendations and used the profiles to implement an exemplary clinical guideline recommendation. CONCLUSIONS: The FHIR implementation guide presented here can be used to directly link the evidence assessment process of evidence-based guideline recommendation development, i.e. systematic reviews and evidence grading, and the underlying evidence from primary studies to the resulting guideline recommendations. This not only allows the evidence on which recommendations are based on to be evaluated transparently and critically, but also enables guideline developers to leverage computable evidence in a more direct way to facilitate the generation of computer-interpretable guideline recommendations.

A Framework for the Development of Living Practice Guidelines in Health Care.

BACKGROUND: Living practice guidelines are increasingly being used to ensure that recommendations are responsive to rapidly emerging evidence. OBJECTIVE: To develop a framework that characterizes the processes of development of living practice guidelines in health care. DESIGN: First, 3 background reviews were conducted: a scoping review of methods papers, a review of handbooks of guideline-producing organizations, and an analytic review of selected living practice guidelines. Second, the core team drafted the first version of the framework. Finally, the core team refined the framework through an online survey and online discussions with a multidisciplinary international group of stakeholders. SETTING: International. PARTICIPANTS: Multidisciplinary group of 51 persons who have experience with guidelines. MEASUREMENTS: Not applicable. RESULTS: A major principle of the framework is that the unit of update in a living guideline is the individual recommendation. In addition to providing definitions, the framework addresses several processes. The planning process should address the organization's adoption of the living methodology as well as each specific guideline project. The production process consists of initiation, maintenance, and retirement phases. The reporting should cover the evidence surveillance time stamp, the outcome of reassessment of the body of evidence (when applicable), and the outcome of revisiting a recommendation (when applicable). The dissemination process may necessitate the use of different venues, including one for formal publication. LIMITATION: This study does not provide detailed or practical guidance for how the described concepts would be best implemented. CONCLUSION: The framework will help guideline developers in planning, producing, reporting, and disseminating living guideline projects. It will also help research methodologists study the processes of living guidelines. PRIMARY FUNDING SOURCE: None.

Making science computable: Developing code systems for statistics, study design, and risk of bias.

The COVID-19 crisis led a group of scientific and informatics experts to accelerate development of an infrastructure for electronic data exchange for the identification, processing, and reporting of scientific findings. The Fast Healthcare Interoperability Resources (FHIR®) standard which is overcoming the interoperability problems in health information exchange was extended to evidence-based medicine (EBM) knowledge with the EBMonFHIR project. A 13-step Code System Development Protocol was created in September 2020 to support global development of terminologies for exchange of scientific evidence. For Step 1, we assembled expert working groups with 55 people from 26 countries by October 2020. For Step 2, we identified 23 commonly used tools and systems for which the first version of code systems will be developed. For Step 3, a total of 368 non-redundant concepts were drafted to become display terms for four code systems (Statistic Type, Statistic Model, Study Design, Risk of Bias). Steps 4 through 13 will guide ongoing development and maintenance of these terminologies for scientific exchange. When completed, the code systems will facilitate identifying, processing, and reporting research results and the reliability of those results. More efficient and detailed scientific communication will reduce cost and burden and improve health outcomes, quality of life, and patient, caregiver, and healthcare professional satisfaction. We hope the achievements reached thus far will outlive COVID-19 and provide an infrastructure to make science computable for future generations. Anyone may join the effort at https://www.gps.health/covid19_knowledge_accelerator.html.

Processes for evidence summarization for patient decision aids: A Delphi consensus study.

BACKGROUND: Patient decision aids (PDAs) should provide evidence-based information so patients can make informed decisions. Yet, PDA developers do not have an agreed-upon process to select, synthesize and present evidence in PDAs. OBJECTIVE: To reach the consensus on an evidence summarization process for PDAs. DESIGN: A two-round modified Delphi survey. SETTING AND PARTICIPANTS: A group of international experts in PDA development invited developers, scientific networks, patient groups and listservs to complete Delphi surveys. DATA COLLECTION: We emailed participants the study description and a link to the online survey. Participants were asked to rate each potential criterion (omit, possible, desirable, essential) and provide qualitative feedback. ANALYSIS: Criteria in each round were retained if rated by >80% of participants as desirable or essential. If two or more participants suggested rewording, reordering or merging, the steering group considered the suggestion. RESULTS: Following two Delphi survey rounds, the evidence summarization process included defining the decision, reporting the processes and policies of the evidence summarization process, assembling the editorial team and managing (collect, manage, report) their conflicts of interest, conducting a systematic search, selecting and appraising the evidence, presenting the harms and benefits in plain language, and describing the method of seeking external review and the plan for updating the evidence (search, selection and appraisal of new evidence). CONCLUSION: A multidisciplinary stakeholder group reached consensus on an evidence summarization process to guide the creation of high-quality PDAs. PATIENT CONTRIBUTION: A patient partner was part of the steering group and involved in the development of the Delphi survey.

RAPADAPTE for rapid guideline development: high-quality clinical guidelines can be rapidly developed with limited resources.

Guideline development is challenging, expensive and labor-intensive. A high-quality guideline with 90 recommendations for breast cancer treatment was developed within 6 months with limited resources in Costa Rica. We describe the experience and propose a process others can use and adapt.The ADAPTE method (using existing guidelines to minimize repeating work that has been done) was used but existing guidelines were not current. The method was extended to use databases that systematically identify, appraise and synthesize evidence for clinical application (DynaMed, EBM Guidelines) to provide current evidence searches and critical appraisal of evidence. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach was used to rate the quality of evidence and the strength of recommendations. Draft recommendations with supporting evidence were provided to panel members for facilitated voting to target panel discussion to areas necessary for reaching consensus.Training panelists in guideline development methodology facilitated rapid consensus development. Extending 'guideline adaptation' to 'evidence database adaptation' was highly effective and efficient. Methods were created to simplify mapping DynaMed evidence ratings to GRADE ratings. Twelve steps are presented to facilitate rapid guideline development and enable further adaptation by others.This is a case report and the RAPADAPTE method was retrospectively derived. Prospective replication and validation will support advances for the guideline development community. If guideline development can be accelerated without compromising validity and relevance of the resulting recommendations this would greatly improve our ability to impact clinical care.

Longitudinal clinical decision support for assessing decisions over time: State-of-the-art and future directions.

Objective: Patients and clinicians rarely experience healthcare decisions as snapshots in time, but clinical decision support (CDS) systems often represent decisions as snapshots. This scoping review systematically maps challenges and facilitators to longitudinal CDS that are applied at two or more timepoints for the same decision made by the same patient or clinician. Methods: We searched Embase, PubMed, and Medline databases for articles describing development, validation, or implementation of patient- or clinician-facing longitudinal CDS. Validated quality assessment tools were used for article selection. Challenges and facilitators to longitudinal CDS are reported according to PRISMA-ScR guidelines. Results: Eight articles met inclusion criteria; each article described a unique CDS. None used entirely automated data entry, none used living guidelines for updating the evidence base or knowledge engine as new evidence emerged during the longitudinal study, and one included formal readiness for change assessments. Seven of eight CDS were implemented and evaluated prospectively. Challenges were primarily related to suboptimal study design (with unique challenges for each study) or user interface. Facilitators included use of randomized trial designs for prospective enrollment, increased CDS uptake during longitudinal exposure, and machine-learning applications that are tailored to the CDS use case. Conclusions: Despite the intuitive advantages of representing healthcare decisions longitudinally, peer-reviewed literature on longitudinal CDS is sparse. Existing reports suggest opportunities to incorporate longitudinal CDS frameworks, automated data entry, living guidelines, and user readiness assessments. Generating best practice guidelines for longitudinal CDS would require a greater depth and breadth of published work and expert opinion.

Reflections on defining a standard for computable expression of scientific knowledge: What teach us Yoda can.

Science advances at a slow pace but can be accelerated with a standard for computable expression of scientific knowledge, more precisely a technical standard for electronic data exchange of machine-interpretable data expressing scientific knowledge. Efforts to achieve this vision include Evidence-Based Medicine on Fast Healthcare Interoperability Resources (EBMonFHIR), COVID-19 Knowledge Accelerator (COKA), Computable Publishing LLC, Scientific Knowledge Accelerator Foundation, and the Fast Evidence Interoperability Resources (FEvIR) Platform. The vision for communicating scientific research results to be instantly found, viewed, sent, received, and incorporated into thousands of system is a Just-in-time Evidence Dissemination and Integration (JEDI) vision. Reflections on JEDI teachings in a fun way helps explain the focus on the "Resource" to make science computable, how a precise specification of evidence changes perspectives, the need to reset standard terminologies, the inclusive nature of efforts to achieve the vision, and the critical demand to realize the vision.

Striking a match between FHIR-based patient data and FHIR-based eligibility criteria.

Inputs and Outputs: The Strike-a-Match Function, written in JavaScript version ES6+, accepts the input of two datasets (one dataset defining eligibility criteria for research studies or clinical decision support, and one dataset defining characteristics for an individual patient). It returns an output signaling whether the patient characteristics are a match for the eligibility criteria. Purpose: Ultimately, such a system will play a "matchmaker" role in facilitating point-of-care recognition of patient-specific clinical decision support. Specifications: The eligibility criteria are defined in HL7 FHIR (version R5) EvidenceVariable Resource JSON structure. The patient characteristics are provided in an FHIR Bundle Resource JSON including one Patient Resource and one or more Observation and Condition Resources which could be obtained from the patient's electronic health record. Application: The Strike-a-Match Function determines whether or not the patient is a match to the eligibility criteria and an Eligibility Criteria Matching Software Demonstration interface provides a human-readable display of matching results by criteria for the clinician or patient to consider. This is the first software application, serving as proof of principle, that compares patient characteristics and eligibility criteria with all data exchanged using HL7 FHIR JSON. An Eligibility Criteria Matching Software Library at https://fevir.net/110192 provides a method for sharing functions using the same information model.

Clinical Quality Measures: A Challenge for-and to-Family Physicians.

INTRODUCTION: Improving design, selection and implementation of appropriate clinical quality measures can reduce harms and costs of health care and improve the quality and experience of care delivery. These measures have not been evaluated for appropriateness for use in performance measurement in a systematic, reproducible, and widely accepted manner. METHODS: We defined 10 criteria for evaluation of measure appropriateness in 4 domains: Patient-centeredness of outcomes, specification of population measured and measure detail, reliable evidence that benefits likely outweigh harms, and independence from significant confounders. We applied these criteria to 24 measures under consideration for statewide use in Massachusetts in public and private incentive-based programs. We appraised each measure as Appropriate or Not Appropriate for such use. RESULTS: We rated 15 measures as Appropriate (62.5%). Three measures (12.5%) were considered Appropriate only if applied at a system level but not for patient-provider assessment and 6 measures (25%) were rated Not Appropriate. Reasons for designation as "Not Appropriate" included benefits not clearly outweighing harms, lack of preservation of patient autonomy, inappropriate specification of population and measure detail, confounding by locus of control, and confounding by social determinants of health. CONCLUSIONS: Using this consensus-driven, 10-criteria methodology we were able to evaluate appropriateness of clinical quality measures. This methodology may improve measure design and inform selection of the most appropriate measures for use in quality measurement, financial incentives, and reporting.

Categorizing metadata to help mobilize computable biomedical knowledge.

INTRODUCTION: Computable biomedical knowledge artifacts (CBKs) are digital objects conveying biomedical knowledge in machine-interpretable structures. As more CBKs are produced and their complexity increases, the value obtained from sharing CBKs grows. Mobilizing CBKs and sharing them widely can only be achieved if the CBKs are findable, accessible, interoperable, reusable, and trustable (FAIR+T). To help mobilize CBKs, we describe our efforts to outline metadata categories to make CBKs FAIR+T. METHODS: We examined the literature regarding metadata with the potential to make digital artifacts FAIR+T. We also examined metadata available online today for actual CBKs of 12 different types. With iterative refinement, we came to a consensus on key categories of metadata that, when taken together, can make CBKs FAIR+T. We use subject-predicate-object triples to more clearly differentiate metadata categories. RESULTS: We defined 13 categories of CBK metadata most relevant to making CBKs FAIR+T. Eleven of these categories (type, domain, purpose, identification, location, CBK-to-CBK relationships, technical, authorization and rights management, provenance, evidential basis, and evidence from use metadata) are evident today where CBKs are stored online. Two additional categories (preservation and integrity metadata) were not evident in our examples. We provide a research agenda to guide further study and development of these and other metadata categories. CONCLUSION: A wide variety of metadata elements in various categories is needed to make CBKs FAIR+T. More work is needed to develop a common framework for CBK metadata that can make CBKs FAIR+T for all stakeholders.

Developing trustworthy recommendations as part of an urgent response (1-2 weeks): a GRADE concept paper.

OBJECTIVES: The aim of this study is to propose an approach for developing trustworthy recommendations as part of urgent responses (1-2 week) in the clinical, public health, and health systems fields. STUDY DESIGN AND SETTING: We conducted a review of the literature, outlined a draft approach, refined the concept through iterative discussions, a workshop by the Grading of Recommendations Assessment, Development and Evaluation Rapid Guidelines project group, and obtained feedback from the larger Grading of Recommendations Assessment, Development and Evaluation working group. RESULTS: A request for developing recommendations within 2 week is the usual trigger for an urgent response. Although the approach builds on the general principles of trustworthy guideline development, we highlight the following steps: (1) assess the level of urgency; (2) assess feasibility; (3) set up the organizational logistics; (4) specify the question(s); (5) collect the information needed; (6) assess the adequacy of identified information; (7) develop the recommendations using one of the 4 potential approaches: adopt existing recommendations, adapt existing recommendations, develop new recommendations using existing adequate systematic review, or develop new recommendations using expert panel input; and (8) consider an updating plan. CONCLUSION: An urgent response for developing recommendations requires building a cohesive, skilled, and highly motivated multidisciplinary team with the necessary clinical, scientific, and methodological expertise; adapting to shifting needs; complying with the principles of transparency; and properly managing conflicts of interest.

Review of online evidence-based practice point-of-care information summary providers: response by the publisher of DynaMed.

In response to Banzi's et al review of online evidence-based practice point-of-care resources published in the Journal of Medical Internet Research, the publisher of DynaMed clarifies his evidence-based methodology.

Thrombolysis with alteplase 3-4.5 hours after acute ischaemic stroke: trial reanalysis adjusted for baseline imbalances.

OBJECTIVES: Alteplase is commonly recommended for acute ischaemic stroke within 4.5 hours after stroke onset. The Third European Cooperative Acute Stroke Study (ECASS III) is the only trial reporting statistically significant efficacy for clinical outcomes for alteplase use 3-4.5 hours after stroke onset. However, baseline imbalances in history of prior stroke and stroke severity score may confound this apparent finding of efficacy. We reanalysed the ECASS III trial data adjusting for baseline imbalances to determine the robustness or sensitivity of the efficacy estimates. DESIGN: Reanalysis of randomised placebo-controlled trial. We obtained access to the ECASS III trial data and replicated the previously reported analyses to confirm our understanding of the data. We adjusted for baseline imbalances using multivariable analyses and stratified analyses and performed sensitivity analysis for missing data. SETTING: Emergency care. PARTICIPANTS: 821 adults with acute ischaemic stroke who could be treated 3-4.5 hours after symptom onset. INTERVENTIONS: Intravenous alteplase (0.9 mg/kg of body weight) or placebo. MAIN OUTCOME MEASURES: The original primary efficacy outcome was modified Rankin Scale (mRS) score 0 or 1 (ie, being alive without any disability) and the original secondary efficacy outcome was a global outcome based on a composite of functional end points, both at 90 days. Adjusted analyses were only reported for the primary efficacy outcome and the original study protocol did not specify methods for adjusted analyses. Our adjusted reanalysis included these outcomes, symptom-free status (mRS 0), dependence-free status (mRS 0-2), mortality (mRS 6) and change across the mRS 0-6 spectrum at 90 days; and mortality and symptomatic intracranial haemorrhage at 7 days. RESULTS: We replicated previously reported unadjusted analyses but discovered they were based on a modified interpretation of the National Institutes of Health Stroke Scale (NIHSS) score. The secondary efficacy outcome was no longer significant using the original NIHSS score. Previously reported adjusted analyses could only be replicated with significant effects for the primary efficacy outcome by using statistical approaches not reported in the trial protocol or statistical analysis plan. In analyses adjusting for baseline imbalances, all efficacy outcomes were not significant, but increases in symptomatic intracranial haemorrhage remained significant. CONCLUSIONS: Reanalysis of the ECASS III trial data with multiple approaches adjusting for baseline imbalances does not support any significant benefits and continues to support harms for the use of alteplase 3-4.5 hours after stroke onset. Clinicians, patients and policymakers should reconsider interpretations and decisions regarding management of acute ischaemic stroke that were based on ECASS III results. TRIAL REGISTRATION NUMBER: NCT00153036.