Ten simple rules to make computable knowledge shareable and reusable.

Computable biomedical knowledge (CBK) is: "the result of an analytic and/or deliberative process about human health, or affecting human health, that is explicit, and therefore can be represented and reasned upon using logic, formal standards, and mathematical approaches." Representing biomedical knowledge in a machine-interpretable, computable form increases its ability to be discovered, accessed, understood, and deployed. Computable knowledge artifacts can greatly advance the potential for implementation, reproducibility, or extension of the knowledge by users, who may include practitioners, researchers, and learners. Enriching computable knowledge artifacts may help facilitate reuse and translation into practice. Following the examples of 10 Simple Rules papers for scientific code, software, and applications, we present 10 Simple Rules intended to make shared computable knowledge artifacts more useful and reusable. These rules are mainly for researchers and their teams who have decided that sharing their computable knowledge is important, who wish to go beyond simply describing results, algorithms, or models via traditional publication pathways, and who want to both make their research findings more accessible, and to help others use their computable knowledge. These rules are roughly organized into 3 categories: planning, engineering, and documentation. Finally, while many of the following examples are of computable knowledge in biomedical domains, these rules are generalizable to computable knowledge in any research domain.

Partnership development of the COVID-19 Front Door: a best evidence resource.

This project describes the creation of a single searchable resource during the pandemic, called the COVID-19 Best Evidence Front Door, with a primary goal of providing direct access to high-quality meta-analyses, literature syntheses, and clinical guidelines from a variety of trusted sources. The Front Door makes relevant evidence findable and accessible with a single search to aggregated evidence-based resources, optimizing time, discovery, and improved access to quality scientific evidence while reducing the burden of frontline health care providers and other knowledge-seekers in needing to separately identify, locate, and explore multiple websites.

A learning pharmacy practice enabled by the pharmacists' patient care process.

To address the Quintuple Aim of health care improvement, the profession of pharmacy is on the verge of a practice transformation that incorporates continuous learning from medication-related data into existing clinical and dispensing roles. The pharmacists' patient care process (PPCP) enables a learning pharmacy practice through the systematic and standardized collection of real-world medication-related data from pharmacists' patient care activities. A learning pharmacy practice continually generates data-powered discoveries as a byproduct of PPCP interactions. In turn, these discoveries improve our medication knowledge while upgrading our predictive powers, thus helping all people achieve optimal health outcomes. Establishing a practice management system connected to the PPCP means that data are generated from every PPCP interaction, combined with existing data, and analyzed by teams of pharmacists and data scientists. The resulting new knowledge is then incorporated into all future PPCP interactions in the form of predictions coupled to actionable advice. The primary purpose of a learning pharmacy practice is to combine the power of predictive modeling with evidence-based best practices to achieve and sustain population-level health improvements. This purpose is achieved by systematically optimizing individual medication use in an equitable manner on a global scale.

Using name overlap analysis to understand medication name search safety.

PURPOSE: To examine the degree of left-to-right character overlap in medication names as they appear in real-world computer systems. METHODS: We programmed a computer to create and automatically analyze left-to-right character overlap in names appearing on 20,020 lists of real-world medication names. The lists varied in length from 100 to 500 medication names and were created by randomly drawing names from a pool of 2,249 medication names extracted from an operating medication use system database. RESULTS: Overall maximum left-to-right character overlap varied in lists of 100 to 500 medication names from 4 to 29 characters (mode of 14 characters). For a small subset of names for high-alert medications that must never be administered in error, overall maximum left-to-right character overlap varied from 3 to 10 characters (mode of 6 characters). Further, for users searching for medications by name in computer systems, the keystrokes that do the most work to disambiguate medication names on a list are always the initial few keystrokes. CONCLUSION: Medication name left-to-right character overlap on lists of names searched ranges widely. Instead of requiring all users to type a set number of characters when searching for medications by name, search safety can potentially be improved by upgrading computer systems to dynamically respond to each keystroke entered. Using incremental dynamic search, searchers would often be able to type fewer than 5 characters to isolate a single medication by name but would sometimes have to type many more than 5 characters to do so.

Advancing maturity modeling for precision oncology.

Introduction: This study aimed to map the maturity of precision oncology as an example of a Learning Health System by understanding the current state of practice, tools and informatics, and barriers and facilitators of maturity. Methods: We conducted semi-structured interviews with 34 professionals (e.g., clinicians, pathologists, and program managers) involved in Molecular Tumor Boards (MTBs). Interviewees were recruited through outreach at 3 large academic medical centers (AMCs) (n = 16) and a Next Generation Sequencing (NGS) company (n = 18). Interviewees were asked about their roles and relationships with MTBs, processes and tools used, and institutional practices. The interviews were then coded and analyzed to understand the variation in maturity across the evolving field of precision oncology. Results: The findings provide insight into the present level of maturity in the precision oncology field, including the state of tooling and informatics within the same domain, the effects of the critical environment on overall maturity, and prospective approaches to enhance maturity of the field. We found that maturity is relatively low, but continuing to evolve, across these dimensions due to the resource-intensive and complex sociotechnical infrastructure required to advance maturity of the field and to fully close learning loops. Conclusion: Our findings advance the field by defining and contextualizing the current state of maturity and potential future strategies for advancing precision oncology, providing a framework to examine how learning health systems mature, and furthering the development of maturity models with new evidence.

A facility-level self-assessment of Autonomous Pharmacy Framework levels.

PURPOSE: The objective of this study was to understand at what level of the Autonomous Pharmacy Framework facilities are operating, in terms of the current state of data collection and analysis in the medication-use process, and to gather insights about systems integration and automation use. METHODS: The Autonomous Pharmacy Advisory Board, a group of chief pharmacy officers and operational leaders, developed a self-assessment instrument based on the previously published Autonomous Pharmacy Framework, made the self-assessment instrument available via the internet, and reviewed respondents' self-reported results. The data collection period for the survey started in March of 2021 and ended in January of 2023. RESULTS: A total of 119 facility-level self-assessments were completed and analyzed. On a scale of 1 to 5, where 1 represented little or no data-driven automation with lots of manual tasks and 5 represented the utmost data-driven automation with few manual tasks, the average overall facility-level score was 2.77 (range, 1.38-4.41). Results revealed slight variance by facility bed capacity. Much more variation was found in the degrees to which individual facilities have automated core processes like inventory management, intravenous medication preparation, and financial reporting. CONCLUSION: As a baseline, this automation-focused facility self-assessment suggests that for essentially all health-system pharmacy facilities and their larger organizations, a substantial body of work needs to be done to further develop and upgrade technology and practice in tandem, greatly expand data collection and analysis, and thereby achieve better operational, financial, and clinical outcomes. Significant advancements are needed to arrive at the highly reliable, highly automated, data-driven medication-use process involving few repetitive manual tasks envisioned in the Autonomous Pharmacy Framework.

CBK model composition using paired web services and executable functions: A demonstration for individualizing preventive services.

Introduction: Learning health systems are challenged to combine computable biomedical knowledge (CBK) models. Using common technical capabilities of the World Wide Web (WWW), digital objects called Knowledge Objects, and a new pattern of activating CBK models brought forth here, we aim to show that it is possible to compose CBK models in more highly standardized and potentially easier, more useful ways. Methods: Using previously specified compound digital objects called Knowledge Objects, CBK models are packaged with metadata, API descriptions, and runtime requirements. Using open-source runtimes and a tool we developed called the KGrid Activator, CBK models can be instantiated inside runtimes and made accessible via RESTful APIs by the KGrid Activator. The KGrid Activator then serves as a gateway and provides a means to interconnect CBK model outputs and inputs, thereby establishing a CBK model composition method. Results: To demonstrate our model composition method, we developed a complex composite CBK model from 42 CBK submodels. The resulting model called CM-IPP is used to compute life-gain estimates for individuals based their personal characteristics. Our result is an externalized, highly modularized CM-IPP implementation that can be distributed and made runnable in any common server environment. Discussion: CBK model composition using compound digital objects and the distributed computing technologies is feasible. Our method of model composition might be usefully extended to bring about large ecosystems of distinct CBK models that can be fitted and re-fitted in various ways to form new composites. Remaining challenges related to the design of composite models include identifying appropriate model boundaries and organizing submodels to separate computational concerns while optimizing reuse potential. Conclusion: Learning health systems need methods for combining CBK models from a variety of sources to create more complex and useful composite models. It is feasible to leverage Knowledge Objects and common API methods in combination to compose CBK models into complex composite models.

A Scalable Service to Improve Health Care Quality Through Precision Audit and Feedback: Proposal for a Randomized Controlled Trial.

BACKGROUND: Health care delivery organizations lack evidence-based strategies for using quality measurement data to improve performance. Audit and feedback (A&F), the delivery of clinical performance summaries to providers, demonstrates the potential for large effects on clinical practice but is currently implemented as a blunt one size fits most intervention. Each provider in a care setting typically receives a performance summary of identical metrics in a common format despite the growing recognition that precisionizing interventions hold significant promise in improving their impact. A precision approach to A&F prioritizes the display of information in a single metric that, for each recipient, carries the highest value for performance improvement, such as when the metric's level drops below a peer benchmark or minimum standard for the first time, thereby revealing an actionable performance gap. Furthermore, precision A&F uses an optimal message format (including framing and visual displays) based on what is known about the recipient and the intended gist meaning being communicated to improve message interpretation while reducing the cognitive processing burden. Well-established psychological principles, frameworks, and theories form a feedback intervention knowledge base to achieve precision A&F. From an informatics perspective, precision A&F requires a knowledge-based system that enables mass customization by representing knowledge configurable at the group and individual levels. OBJECTIVE: This study aims to implement and evaluate a demonstration system for precision A&F in anesthesia care and to assess the effect of precision feedback emails on care quality and outcomes in a national quality improvement consortium. METHODS: We propose to achieve our aims by conducting 3 studies: a requirements analysis and preferences elicitation study using human-centered design and conjoint analysis methods, a software service development and implementation study, and a cluster randomized controlled trial of a precision A&F service with a concurrent process evaluation. This study will be conducted with the Multicenter Perioperative Outcomes Group, a national anesthesia quality improvement consortium with >60 member hospitals in >20 US states. This study will extend the Multicenter Perioperative Outcomes Group quality improvement infrastructure by using existing data and performance measurement processes. RESULTS: The proposal was funded in September 2021 with a 4-year timeline. Data collection for Aim 1 began in March 2022. We plan for a 24-month trial timeline, with the intervention period of the trial beginning in March 2024. CONCLUSIONS: The proposed aims will collectively demonstrate a precision feedback service developed using an open-source technical infrastructure for computable knowledge management. By implementing and evaluating a demonstration system for precision feedback, we create the potential to observe the conditions under which feedback interventions are effective. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): PRR1-10.2196/34990.

Comparing the variability of ingredient, strength, and dose form information from electronic prescriptions with RxNorm drug product descriptions.

OBJECTIVE: To determine the variability of ingredient, strength, and dose form information from drug product descriptions in real-world electronic prescription (e-prescription) data. MATERIALS AND METHODS: A sample of 10 399 324 e-prescriptions from 2019 to 2021 were obtained. Drug product descriptions were analyzed with a named entity extraction model and National Drug Codes (NDCs) were used to get RxNorm Concept Unique Identifiers (RxCUI) via RxNorm. The number of drug product description variants for each RxCUI was determined. Variants identified were compared to RxNorm to determine the extent of matching terminology used. RESULTS: A total of 353 002 unique pairs of drug product descriptions and NDCs were analyzed. The median (1st-3rd quartile) number of variants extracted for each standardized expression in RxNorm, was 3 (2-7) for ingredients, 4 (2-8) for strength, and 41 (11-122) for dosage forms. Of the pairs, 42.35% of ingredients (n = 328 032), 51.23% of strengths (n = 321 706), and 10.60% of dose forms (n = 326 653) used matching terminology, while 16.31%, 24.85%, and 13.05% contained nonmatching terminology, respectively. DISCUSSION: A wide variety of drug product descriptions makes it difficult to determine whether 2 drug product descriptions describe the same drug product (eg, using abbreviations to describe an active ingredient or using different units to represent a concentration). This results in patient safety risks that lead to incorrect drug products being ordered, dispensed, and used by patients. Implementation and use of standardized terminology may reduce these risks. CONCLUSION: Drug product descriptions on real-world e-prescriptions exhibit large variation resulting in unnecessary ambiguity and potential patient safety risks.

Personalized Disease Prevention (PDP): study protocol for a cluster-randomized clinical trial.

BACKGROUND: The US Preventive Services Task Force recommends 25 primary preventive services for middle-aged adults, but it can be difficult to do them all. METHODS: The Personalized Disease Prevention (PDP) cluster-randomized clinical trial will evaluate whether patients and their providers benefit from an evidence-based decision tool to prioritize preventive services based on their potential to improve quality-adjusted life expectancy. The decision tool will be individualized for patient risk factors and available in the electronic health record. This Phase III trial seeks to enroll 60 primary care providers (clusters) and 600 patients aged 40-75 years. Half of providers will be assigned to an intervention to utilize the decision tool with approximately 10 patients each, and half will be assigned to usual care. Mixed-methods follow-up will include collection of preventive care utilization from electronic health records, patient and physician surveys, and qualitative interviews. We hypothesize that quality-adjusted life expectancy will increase by more in patients who receive the intervention, as compared with controls. DISCUSSION: PDP will test a novel, holistic approach to help patients and providers prioritize the delivery of preventive services, based on patient risk factors in the electronic health record. TRIAL REGISTRATION: ClinicalTrials.gov NCT05463887. Registered on July 19, 2022.

Implementation outcomes of the Structured and Codified SIG format in electronic prescription directions.

OBJECTIVE: To determine the extent of implementation, completeness, and accuracy of Structured and Codified SIG (S&C SIG) directions on electronic prescriptions (e-prescriptions). MATERIALS AND METHODS: A retrospective analysis of a random sample of 3.8 million e-prescriptions sent from electronic prescribing (e-prescribing) software to outpatient pharmacies in the United States between 2019 and 2021. Natural language processing was used to identify direction components, including action verb, dose, frequency, route, duration, and indication from free-text directions and were compared to the S&C SIG format. Inductive qualitative analysis of S&C direction identified error types and frequencies for each component. RESULTS: Implementation of the S&C SIG format in e-prescribing software resulted in 32.4% of e-prescriptions transmitted with these standardized directions. Directions using the S&C SIG format contained a greater percentage of each direction component compared to free-text directions, except for the indication component. Structured and codified directions contained quality issues in 10.3% of cases. DISCUSSION: Expanding adoption of more diverse direction terminology for the S&C SIG formats can improve the coverage of directions using the S&C SIG format. Building out e-prescribing software interfaces to include more direction components can improve patient medication use and safety. Quality improvement efforts, such as improving the design of e-prescribing software and auditing for discrepancies, are needed to identify and eliminate implementation-related issues with direction information from the S&C SIG format so that e-prescription directions are always accurately represented. CONCLUSION: Although directions using the S&C SIG format may result in more complete directions, greater adoption of the format and best practices for preventing its incorrect use are necessary.

Expanding Access to CNS-TAP: Design, Development, and Initial Use of a Complex Precision Health Specialty Web App for Neuro-Oncology.

This paper offers a case study to demonstrate how a complex scoring model tool called CNS-TAP, originally created by a neuro-oncology team at one institution, was upgraded and made accessible to a wider audience. In the Results and Discussion, many issues of web app design, development, and sustainability are covered. Overall, we chart a path to expand access to many unique software tools created and needed by today's medical specialists.

Categorizing metadata to help mobilize computable biomedical knowledge.

INTRODUCTION: Computable biomedical knowledge artifacts (CBKs) are digital objects conveying biomedical knowledge in machine-interpretable structures. As more CBKs are produced and their complexity increases, the value obtained from sharing CBKs grows. Mobilizing CBKs and sharing them widely can only be achieved if the CBKs are findable, accessible, interoperable, reusable, and trustable (FAIR+T). To help mobilize CBKs, we describe our efforts to outline metadata categories to make CBKs FAIR+T. METHODS: We examined the literature regarding metadata with the potential to make digital artifacts FAIR+T. We also examined metadata available online today for actual CBKs of 12 different types. With iterative refinement, we came to a consensus on key categories of metadata that, when taken together, can make CBKs FAIR+T. We use subject-predicate-object triples to more clearly differentiate metadata categories. RESULTS: We defined 13 categories of CBK metadata most relevant to making CBKs FAIR+T. Eleven of these categories (type, domain, purpose, identification, location, CBK-to-CBK relationships, technical, authorization and rights management, provenance, evidential basis, and evidence from use metadata) are evident today where CBKs are stored online. Two additional categories (preservation and integrity metadata) were not evident in our examples. We provide a research agenda to guide further study and development of these and other metadata categories. CONCLUSION: A wide variety of metadata elements in various categories is needed to make CBKs FAIR+T. More work is needed to develop a common framework for CBK metadata that can make CBKs FAIR+T for all stakeholders.

A bottom-up approach to creating an ontology for medication indications.

OBJECTIVES: The study sought to learn if it were possible to develop an ontology that would allow the Food and Drug Administration approved indications to be expressed in a manner computable and comparable to what is expressed in an electronic health record. MATERIALS AND METHODS: A random sample of 1177 of the 3000+ extant, distinct medical products (identified by unique new drug application numbers) was selected for investigation. Close manual examination of the indication portion of the labels for these drugs led to the development of a formal model of indications. RESULTS: The model represents each narrative indication as a disjunct of conjuncts of assertions about an individual. A desirable attribute is that each assertion about an individual should be testable without reference to other contextual information about the situation. The logical primitives are chosen from 2 categories (context and conditions) and are linked to an enumeration of uses, such as prevention. We found that more than 99% of approved label indications for treatment or prevention could be so represented. DISCUSSION: While some indications are straightforward to represent, difficulties stem from the need to represent temporal or sequential references. In addition, there is a mismatch of terminologies between what is present in an electronic health record and in the label narrative. CONCLUSIONS: A workable model for formalizing drug indications is possible. Remaining challenges include designing workflow to model narrative label indications for all approved drug products and incorporation of standard vocabularies.

Detecting Potential Medication Selection Errors During Outpatient Pharmacy Processing of Electronic Prescriptions With the RxNorm Application Programming Interface: Retrospective Observational Cohort Study.

BACKGROUND: Medication errors are pervasive. Electronic prescriptions (e-prescriptions) convey secure and computer-readable prescriptions from clinics to outpatient pharmacies for dispensing. Once received, pharmacy staff perform a transcription task to select the medications needed to process e-prescriptions within their dispensing software. Later, pharmacists manually double-check medications selected to fulfill e-prescriptions before dispensing to the patient. Although pharmacist double-checks are mostly effective for catching medication selection mistakes, the cognitive process of medication selection in the computer is still prone to error because of heavy workload, inattention, and fatigue. Leveraging health information technology to identify and recover from medication selection errors can improve patient safety. OBJECTIVE: This study aimed to determine the performance of an automated double-check of pharmacy prescription records to identify potential medication selection errors made in outpatient pharmacies with the RxNorm application programming interface (API). METHODS: We conducted a retrospective observational analysis of 537,710 pairs of e-prescription and dispensing records from a mail-order pharmacy for the period January 2017 to October 2018. National Drug Codes (NDCs) for each pair were obtained from the National Library of Medicine's (NLM's) RxNorm API. The API returned RxNorm concept unique identifier (RxCUI) semantic clinical drug (SCD) identifiers associated with every NDC. The SCD identifiers returned for the e-prescription NDC were matched against the corresponding SCD identifiers from the pharmacy dispensing record NDC. An error matrix was created based on the hand-labeling of mismatched SCD pairs. Performance metrics were calculated for the e-prescription-to-dispensing record matching algorithm for both total pairs and unique pairs of NDCs in these data. RESULTS: We analyzed 527,881 e-prescription and pharmacy dispensing record pairs. Four clinically significant cases of mismatched RxCUI identifiers were detected (ie, three different ingredient selections and one different strength selection). A total of 546 less significant cases of mismatched RxCUIs were found. Nearly all of the NDC pairs had matching RxCUIs (28,787/28,817, 99.90%-525,270/527,009, 99.67%). The RxNorm API had a sensitivity of 1, a false-positive rate of 0.00104 to 0.00312, specificity of 0.99896 to 0.99688, precision of 0.00727 to 0.04255, and F1 score of 0.01444 to 0.08163. We found 872 pairs of records without an RxCUI. CONCLUSIONS: The NLM's RxNorm API can perform an independent and automatic double-check of correct medication selection to verify e-prescription processing at outpatient pharmacies. RxNorm has near-comprehensive coverage of prescribed medications and can be used to recover from medication selection errors. In the future, tools such as this may be able to perform automated verification of medication selection accurately enough to free pharmacists from having to perform manual double-checks of the medications selected within pharmacy dispensing software to fulfill e-prescriptions.

Engaging Pharmacists to Crowdsource a Fine-grained Medication Risk Scale: An Initial Measurement Study Using Paired Comparisons of Medications.

A coarse classification of medications into two risk categories, one for high-risk medications and one for all others, allows people to focus safety improvement work on medications that carry the highest risks of harm. However, such coarse categorization does not distinguish the relative risk of harm for the majority of medications. To begin to develop a more fine-grained measurement scale for the relative risk of harm spanning many medications, we performed an experiment with 18 practicing pharmacists. Each pharmacist-participant made 210 paired comparisons of 21 commonly prescribed medications to reveal a subjective scale of perceived medication worrisomeness (PMW). Statistical analyses of their collective judgments of medication pairs differentiated five levels of PMW. This study illuminates one path towards a fine-grained medication risk scale based on PMW. It also shows how the method of paired comparisons can be used to remotely crowdsource expert knowledge in support of learning health systems.

Initial study of clinical pharmacy work prioritization tools.

PURPOSE: The development, implementation, and validity of clinical pharmacy work prioritization tools (CPWPTs) were analyzed. METHODS: Informatics pharmacists were contacted and asked to participate in 30-minute teleconference interviews, as they have primary responsibility for the technical configuration of CPWPTs. A total of 19 respondents participated in the survey. The qualitative data collected encompassed the thoughts and ideas of practicing informatics pharmacists who are knowledgeable about the design, configuration, management, and use of CPWPTs. In addition to capturing their thoughts and ideas with open-ended questions, demographic data were collected, as was information about the sites where respondents worked and the CPWPTs they used. RESULTS: Most of the CPWPTs were built into existing electronic health record platforms. There was considerable variation among the prioritization factors used at each site. The most commonly identified categories of prioritization factors were patient-specific factors, therapeutic classes of medications, and potential pharmacist interventions. All respondents reported that the prioritized tasks generated by their CPWPTs were examined for face validity. Of the 19 respondents, only 4 reported that the priorities generated by their CPWPT had been empirically validated in some way. Qualitative data analysis revealed that informatics pharmacists have 5 general perceptions about CPWPT factors, validation, and use in practice: (1) mirroring practice, (2) pharmacist consensus-based design, (3) complexity of logic, (4) tension between task-oriented and patient-centric approaches to practice, and (5) comfort from tracking tasks to completion. CONCLUSION: Early CPWPTs vary significantly in their prioritization factors. These tools partially reflect the scope of clinical pharmacy practice at the sites where they are used.

The Knowledge Object Reference Ontology (KORO): A formalism to support management and sharing of computable biomedical knowledge for learning health systems.

INTRODUCTION: Health systems are challenged by care underutilization, overutilization, disparities, and related harms. One problem is a multiyear latency between discovery of new best practice knowledge and its widespread adoption. Decreasing this latency requires new capabilities to better manage and more rapidly share biomedical knowledge in computable forms. Knowledge objects package machine-executable knowledge resources in a way that easily enables knowledge as a service. To help improve knowledge management and accelerate knowledge sharing, the Knowledge Object Reference Ontology (KORO) defines what knowledge objects are in a formal way. METHODS: Development of KORO began with identification of terms for classes of entities and for properties. Next, we established a taxonomical hierarchy of classes for knowledge objects and their parts. Development continued by relating these parts via formally defined properties. We evaluated the logical consistency of KORO and used it to answer several competency questions about parthood. We also applied it to guide knowledge object implementation. RESULTS: As a realist ontology, KORO defines what knowledge objects are and provides details about the parts they have and the roles they play. KORO provides sufficient logic to answer several basic but important questions about knowledge objects competently. KORO directly supports creators of knowledge objects by providing a formal model for these objects. CONCLUSION: KORO provides a formal, logically consistent ontology about knowledge objects and their parts. It exists to help make computable biomedical knowledge findable, accessible, interoperable, and reusable. KORO is currently being used to further develop and improve computable knowledge infrastructure for learning health systems.