Ten simple rules to make computable knowledge shareable and reusable.

Computable biomedical knowledge (CBK) is: "the result of an analytic and/or deliberative process about human health, or affecting human health, that is explicit, and therefore can be represented and reasned upon using logic, formal standards, and mathematical approaches." Representing biomedical knowledge in a machine-interpretable, computable form increases its ability to be discovered, accessed, understood, and deployed. Computable knowledge artifacts can greatly advance the potential for implementation, reproducibility, or extension of the knowledge by users, who may include practitioners, researchers, and learners. Enriching computable knowledge artifacts may help facilitate reuse and translation into practice. Following the examples of 10 Simple Rules papers for scientific code, software, and applications, we present 10 Simple Rules intended to make shared computable knowledge artifacts more useful and reusable. These rules are mainly for researchers and their teams who have decided that sharing their computable knowledge is important, who wish to go beyond simply describing results, algorithms, or models via traditional publication pathways, and who want to both make their research findings more accessible, and to help others use their computable knowledge. These rules are roughly organized into 3 categories: planning, engineering, and documentation. Finally, while many of the following examples are of computable knowledge in biomedical domains, these rules are generalizable to computable knowledge in any research domain.

Barriers to the Initiation of Telecommunicator-CPR during 9-1-1 Out-of-Hospital Cardiac Arrest Calls: A Qualitative Study.

INTRODUCTION: Fewer than 10% of individuals who suffer out-of-hospital cardiac arrest (OHCA) survive with good neurologic function. Bystander CPR more than doubles the chance of survival, and telecommunicator-CPR (T-CPR) during a 9-1-1 call substantially improves the frequency of bystander CPR. OBJECTIVE: We examined the barriers to initiation of T-CPR. METHODS: We analyzed the 9-1-1 call audio from 65 EMS-treated OHCAs from a single US 9-1-1 dispatch center. We initially conducted a thematic analysis aimed at identifying barriers to the initiation of T-CPR. We then conducted a conversation analysis that examined the interactions between telecommunicators and bystanders during the recognition phase (i.e., consciousness and normal breathing). RESULTS: We identified six process themes related to barriers, including incomplete or delayed recognition assessment, delayed repositioning, communication gaps, caller emotional distress, nonessential questions and assessments, and caller refusal, hesitation, or inability to act. We identified three suboptimal outcomes related to arrest recognition and delivery of chest compressions, which are missed OHCA identification, delayed OHCA identification and treatment, and compression instructions not provided following OHCA identification. A primary theme observed during missed OHCA calls was incomplete or delayed recognition assessment and included failure to recognize descriptors indicative of agonal breathing (e.g., "snoring", "slow") or to confirm that breathing was effective in an unconscious victim. CONCLUSIONS: We observed that modifiable barriers identified during 9-1-1 calls where OHCA was missed, or treatment was delayed, were often related to incomplete or delayed recognition assessment. Repositioning delays were a common barrier to the initiation of chest compressions.

Partnership development of the COVID-19 Front Door: a best evidence resource.

This project describes the creation of a single searchable resource during the pandemic, called the COVID-19 Best Evidence Front Door, with a primary goal of providing direct access to high-quality meta-analyses, literature syntheses, and clinical guidelines from a variety of trusted sources. The Front Door makes relevant evidence findable and accessible with a single search to aggregated evidence-based resources, optimizing time, discovery, and improved access to quality scientific evidence while reducing the burden of frontline health care providers and other knowledge-seekers in needing to separately identify, locate, and explore multiple websites.

Variation in Physicians' Electronic Health Record Documentation and Potential Patient Harm from That Variation.

BACKGROUND: Physician-to-physician variation in electronic health record (EHR) documentation not driven by patients' clinical status could be harmful. OBJECTIVE: Measure variation in completion of common clinical documentation domains. Identify perceived causes and effects of variation and strategies to mitigate negative effects. DESIGN: Sequential, explanatory, mixed methods using log data from a commercial EHR vendor and semi-structured interviews with outpatient primary care practices. PARTICIPANTS: Quantitative: 170,332 encounters led by 809 physicians in 237 practices. Qualitative: 40 interviewees in 10 practices. MAIN MEASURES: Interquartile range (IQR) of the proportion of encounters in which a physician completed documentation, for each documentation category. Multilevel linear regression measured the proportion of variation at the physician level. KEY RESULTS: Five clinical documentation categories had substantial and statistically significant (p < 0.001) variation at the physician level after accounting for state, organization, and practice levels: (1) discussing results (IQR = 50.8%, proportion of variation explained by physician level = 78.1%); (2) assessment and diagnosis (IQR = 60.4%, physician-level variation = 76.0%); (3) problem list (IQR = 73.1%, physician-level variation = 70.1%); (4) review of systems (IQR = 62.3%, physician-level variation = 67.7%); and (5) social history (IQR = 53.3%, physician-level variation = 62.2%). Drivers of variation from interviews included user preferences and EHR designs with multiple places to record similar information. Variation was perceived to create documentation inefficiencies and risk patient harm due to missed or misinterpreted information. Mitigation strategies included targeted user training during EHR implementation and practice meetings focused on documentation standardization. CONCLUSIONS: Physician-to-physician variation in EHR documentation impedes effective and safe use of EHRs, but there are potential strategies to mitigate negative consequences.

Mind the Gap: Putting Evidence into Practice in the Era of Learning Health Systems.

Due to the increasing amount of available published evidence and the continual need to apply and update evidence in practice, we propose a shift in the way evidence generated by learning health systems can be integrated into more traditional evidence reviews. This paper discusses two main mechanisms to close the evidence-to-practice gap: (1) integrating Learning Health System (LHS) results with existing systematic review evidence and (2) providing this combined evidence in a standardized, computable data format. We believe these efforts will better inform practice, thereby improving individual and population health.

Educating medical students in the era of ubiquitous information.

Health care around the world is going digital. This inexorable trend will result in: (1) routine documentation of care in digital form and emerging national infrastructures for sharing data that allow progress toward a learning health system; and (2) a biomedical "knowledge cloud" that is fully integrated into practice environments and accessible to both providers and consumers of healthcare. Concurrently, medical students will be complete digital natives who have literally grown up with the Internet and will enter practice early in the next decade when the projected changes in practice approach maturity. This essay describes three competencies linked to this evolving information environment-(1) knowing what you do and don't know, (2) ability to ask a good question, and (3) skills in evaluating and weighing evidence-and suggests educational approaches to promote student mastery of each competency. Shifting medical education to address these competencies will call into question many current methods but may be essential to fully prepare trainees for optimal practice in the future.

Exploring Anesthesia Provider Preferences for Precision Feedback: Preference Elicitation Study.

Background: Health care professionals must learn continuously as a core part of their work. As the rate of knowledge production in biomedicine increases, better support for health care professionals' continuous learning is needed. In health systems, feedback is pervasive and is widely considered to be essential for learning that drives improvement. Clinical quality dashboards are one widely deployed approach to delivering feedback, but engagement with these systems is commonly low, reflecting a limited understanding of how to improve the effectiveness of feedback about health care. When coaches and facilitators deliver feedback for improving performance, they aim to be responsive to the recipient's motivations, information needs, and preferences. However, such functionality is largely missing from dashboards and feedback reports. Precision feedback is the delivery of high-value, motivating performance information that is prioritized based on its motivational potential for a specific recipient, including their needs and preferences. Anesthesia care offers a clinical domain with high-quality performance data and an abundance of evidence-based quality metrics. Objective: The objective of this study is to explore anesthesia provider preferences for precision feedback. Methods: We developed a test set of precision feedback messages with balanced characteristics across 4 performance scenarios. We created an experimental design to expose participants to contrasting message versions. We recruited anesthesia providers and elicited their preferences through analysis of the content of preferred messages. Participants additionally rated their perceived benefit of preferred messages to clinical practice on a 5-point Likert scale. Results: We elicited preferences and feedback message benefit ratings from 35 participants. Preferences were diverse across participants but largely consistent within participants. Participants' preferences were consistent for message temporality (α=.85) and display format (α=.80). Ratings of participants' perceived benefit to clinical practice of preferred messages were high (mean rating 4.27, SD 0.77). Conclusions: Health care professionals exhibited diverse yet internally consistent preferences for precision feedback across a set of performance scenarios, while also giving messages high ratings of perceived benefit. A "one-size-fits-most approach" to performance feedback delivery would not appear to satisfy these preferences. Precision feedback systems may hold potential to improve support for health care professionals' continuous learning by accommodating feedback preferences.

Early intranasal medication administration in out-of-hospital cardiac arrest: Two randomized simulation trials.

Objective: Intranasal medications have been proposed as adjuncts to out-of-hospital cardiac arrest (OHCA) care. We sought to quantify the effects of intranasal medication administration (INMA) in OHCA workflows. Methods: We conducted separate randomized OHCA simulation trials with lay rescuers (LRs) and first responders (FRs). Participants were randomized to groups performing hands-only cardiopulmonary resuscitation (CPR)/automated external defibrillator with or without INMA during the second analysis phase. Time to compression following the second shock (CPR2) was the primary outcome and compression quality (chest compression rate (CCR) and fraction (CCF)) was the secondary outcome. We fit linear regression models adjusted for CPR training in the LR group and service years in the FR group. Results: Among LRs, INMA was associated with a significant increase in CPR2 (mean diff. 44.1 s, 95% CI: 14.9, 73.3), which persisted after adjustment (p = 0.005). We observed a significant decrease in CCR (INMA 95.1 compressions per min (cpm) vs control 104.2 cpm, mean diff. -9.1 cpm, 95% CI -16.6, -1.6) and CCF (INMA 62.4% vs control 69.8%, mean diff. -7.5%, 95% CI -12.0, -2.9). Among FRs, we found no significant CPR2 delays (mean diff. -2.1 s, 95% CI -15.9, 11.7), which persisted after adjustment (p = 0.704), or difference in quality (CCR INMA 115.5 cpm vs control 120.8 cpm, mean diff. -5.3 cpm, 95% CI -12.6, 2.0; CCF INMA 79.6% vs control 81.2% mean diff. -1.6%, 95% CI -7.4, 4.3%). Conclusions: INMA in LR resuscitation was associated with diminished resuscitation performance. INMA by FR did not impede key times or quality.

CBK model composition using paired web services and executable functions: A demonstration for individualizing preventive services.

Introduction: Learning health systems are challenged to combine computable biomedical knowledge (CBK) models. Using common technical capabilities of the World Wide Web (WWW), digital objects called Knowledge Objects, and a new pattern of activating CBK models brought forth here, we aim to show that it is possible to compose CBK models in more highly standardized and potentially easier, more useful ways. Methods: Using previously specified compound digital objects called Knowledge Objects, CBK models are packaged with metadata, API descriptions, and runtime requirements. Using open-source runtimes and a tool we developed called the KGrid Activator, CBK models can be instantiated inside runtimes and made accessible via RESTful APIs by the KGrid Activator. The KGrid Activator then serves as a gateway and provides a means to interconnect CBK model outputs and inputs, thereby establishing a CBK model composition method. Results: To demonstrate our model composition method, we developed a complex composite CBK model from 42 CBK submodels. The resulting model called CM-IPP is used to compute life-gain estimates for individuals based their personal characteristics. Our result is an externalized, highly modularized CM-IPP implementation that can be distributed and made runnable in any common server environment. Discussion: CBK model composition using compound digital objects and the distributed computing technologies is feasible. Our method of model composition might be usefully extended to bring about large ecosystems of distinct CBK models that can be fitted and re-fitted in various ways to form new composites. Remaining challenges related to the design of composite models include identifying appropriate model boundaries and organizing submodels to separate computational concerns while optimizing reuse potential. Conclusion: Learning health systems need methods for combining CBK models from a variety of sources to create more complex and useful composite models. It is feasible to leverage Knowledge Objects and common API methods in combination to compose CBK models into complex composite models.

Applying AI and Guidelines to Assist Medical Students in Recognizing Patients With Heart Failure: Protocol for a Randomized Trial.

BACKGROUND: The integration of artificial intelligence (AI) into clinical practice is transforming both clinical practice and medical education. AI-based systems aim to improve the efficacy of clinical tasks, enhancing diagnostic accuracy and tailoring treatment delivery. As it becomes increasingly prevalent in health care for high-quality patient care, it is critical for health care providers to use the systems responsibly to mitigate bias, ensure effective outcomes, and provide safe clinical practices. In this study, the clinical task is the identification of heart failure (HF) prior to surgery with the intention of enhancing clinical decision-making skills. HF is a common and severe disease, but detection remains challenging due to its subtle manifestation, often concurrent with other medical conditions, and the absence of a simple and effective diagnostic test. While advanced HF algorithms have been developed, the use of these AI-based systems to enhance clinical decision-making in medical education remains understudied. OBJECTIVE: This research protocol is to demonstrate our study design, systematic procedures for selecting surgical cases from electronic health records, and interventions. The primary objective of this study is to measure the effectiveness of interventions aimed at improving HF recognition before surgery, the second objective is to evaluate the impact of inaccurate AI recommendations, and the third objective is to explore the relationship between the inclination to accept AI recommendations and their accuracy. METHODS: Our study used a 3 × 2 factorial design (intervention type × order of prepost sets) for this randomized trial with medical students. The student participants are asked to complete a 30-minute e-learning module that includes key information about the intervention and a 5-question quiz, and a 60-minute review of 20 surgical cases to determine the presence of HF. To mitigate selection bias in the pre- and posttests, we adopted a feature-based systematic sampling procedure. From a pool of 703 expert-reviewed surgical cases, 20 were selected based on features such as case complexity, model performance, and positive and negative labels. This study comprises three interventions: (1) a direct AI-based recommendation with a predicted HF score, (2) an indirect AI-based recommendation gauged through the area under the curve metric, and (3) an HF guideline-based intervention. RESULTS: As of July 2023, 62 of the enrolled medical students have fulfilled this study's participation, including the completion of a short quiz and the review of 20 surgical cases. The subject enrollment commenced in August 2022 and will end in December 2023, with the goal of recruiting 75 medical students in years 3 and 4 with clinical experience. CONCLUSIONS: We demonstrated a study protocol for the randomized trial, measuring the effectiveness of interventions using AI and HF guidelines among medical students to enhance HF recognition in preoperative care with electronic health record data. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/49842.

Factors Impacting Treatment of Out-of-Hospital Cardiac Arrest: A Qualitative Study of Emergency Responders.

Background Of the more than 250 000 emergency medical services-treated out-of-hospital cardiac arrests that occur each year in the United States, only about 8% survive to hospital discharge with good neurologic function. Treatment for out-of-hospital cardiac arrest involves a system of care that includes complex interactions among multiple stakeholders. Understanding the factors inhibiting optimal care is fundamental to improving outcomes. Methods and Results We conducted group interviews with emergency responders including 911 telecommunicators, law enforcement officers, firefighters, and transporting emergency medical services personnel (ie, emergency medical technicians and paramedics) who responded to the same out-of-hospital cardiac arrest incident. We used the American Heart Association System of Care as the framework for our analysis to identify themes and their contributory factors from these interviews. We identified 5 themes under the structure domain, which included workload, equipment, prehospital communication structure, education and competency, and patient attitudes. In the process domain, 5 themes were identified focusing on preparedness, field response and access to patient, on-scene logistics, background information acquisition, and clinical interventions. We identified 3 system themes including emergency responder culture; community support, education, and engagement; and stakeholder relationships. Three continuous quality improvement themes were identified, which included feedback provision, change management, and documentation. Conclusions We identified structure, process, system, and continuous quality improvement themes that may be leveraged to improve outcomes for out-of-hospital cardiac arrest. Interventions or programs amenable to rapid implementation include improving prearrival communication between agencies, appointing patient care and logistical leadership on-scene, interstakeholder team training, and providing more standardized feedback to all responder groups.

Evaluating Completeness of Discrete Data on Physical Functioning for Children With Cerebral Palsy in a Pediatric Rehabilitation Learning Health System.

OBJECTIVE: The purpose of this study was to determine the extent that physical function discrete data elements (DDE) documented in electronic health records (EHR) are complete within pediatric rehabilitation settings. METHODS: A descriptive analysis on completeness of EHR-based DDEs detailing physical functioning for children with cerebral palsy was conducted. Data from an existing pediatric rehabilitation research learning health system data network, consisting of EHR data from 20 care sites in a pediatric specialty health care system, were leveraged. Completeness was calculated for unique data elements, unique outpatient visits, and unique outpatient records. RESULTS: Completeness of physical function DDEs was low across 5766 outpatient records (10.5%, approximately 2 DDEs documented). The DDE for Gross Motor Function Classification System level was available for 21% (n = 3746) outpatient visits and 38% of patient records. Ambulation level was the most frequently documented DDE. Intercept only mixed effects models demonstrated that 21.4% and 45% of the variance in completeness for DDEs and the Gross Motor Function Classification System, respectively, across unique patient records could be attributed to factors at the individual care site level. CONCLUSION: Values of physical function DDEs are missing in designated fields of the EHR infrastructure for pediatric rehabilitation providers. Although completeness appears limited for these DDEs, our observations indicate that data are not missing at random and may be influenced by system-level standards in clinical documentation practices between providers and factors specific to individual care sites. The extent of missing data has significant implications for pediatric rehabilitation quality measurement. More research is needed to understand why discrete data are missing in EHRs and to further elucidate the professional and system-level factors that influence completeness and missingness. IMPACT: Completeness of DDEs reported in this study is limited and presents a significant opportunity to improve documentation and standards to optimize EHR data for learning health system research and quality measurement in pediatric rehabilitation settings.

Assessment of telecommunicator cardiopulmonary resuscitation performance during out-of-hospital cardiac arrest using a standardized tool for audio review.

OBJECTIVE: Telecommunicator cardiopulmonary resuscitation (T-CPR) is a critical component of optimized out-of-hospital cardiac arrest (OHCA) care. We assessed a pilot tool to capture American Heart Association (AHA) T-CPR measures and T-CPR coaching by telecommunicators using audio review. METHODS: Using a pilot tool, we conducted a retrospective review of 911 call audio from 65 emergency medical services-treated out-of-hospital cardiac arrest (OHCA) patients. Data collection included events (e.g., OHCA recognition), time intervals, and coaching quality measures. We calculated summary statistics for all performance and quality measures. RESULTS: Among 65 cases, the patients' mean age was 64.7 years (SD: 14.6) and 17 (26.2%) were women. Telecommunicator recognition occurred in 72% of cases (47/65). Among 18 non-recognized cases, reviewers determined 12 (66%) were not recognizable based on characteristics of the call. Median time-to-recognition was 76 seconds (n = 40; IQR:39-138), while median time-to-first-instructed-compression was 198 seconds (n = 26; IQR:149-233). In 36 cases where coaching was needed, coaching on compression-depth occurred in 27 (75%); -rate in 28 (78%); and chest recoil in 10 (28%) instances. In 30 cases where repositioning was needed, instruction to position the patient's body flat occurred in 18 (60%) instances, on-back in 22 (73%) instances, and on-ground in 22 (73%) instances. CONCLUSIONS: Successful collection of data to calculate AHA T-CPR measures using a pilot tool for audio review revealed performance near AHA benchmarks, although coaching instructions did not occur in many instances. Application of this standardized tool may aid in T-CPR quality review.

Use of electronic health records to support a public health response to the COVID-19 pandemic in the United States: a perspective from 15 academic medical centers.

Our goal is to summarize the collective experience of 15 organizations in dealing with uncoordinated efforts that result in unnecessary delays in understanding, predicting, preparing for, containing, and mitigating the COVID-19 pandemic in the US. Response efforts involve the collection and analysis of data corresponding to healthcare organizations, public health departments, socioeconomic indicators, as well as additional signals collected directly from individuals and communities. We focused on electronic health record (EHR) data, since EHRs can be leveraged and scaled to improve clinical care, research, and to inform public health decision-making. We outline the current challenges in the data ecosystem and the technology infrastructure that are relevant to COVID-19, as witnessed in our 15 institutions. The infrastructure includes registries and clinical data networks to support population-level analyses. We propose a specific set of strategic next steps to increase interoperability, overall organization, and efficiencies.

Reconsidering hospital EHR adoption at the dawn of HITECH: implications of the reported 9% adoption of a "basic" EHR.

OBJECTIVE: In 2009, a prominent national report stated that 9% of US hospitals had adopted a "basic" electronic health record (EHR) system. This statistic was widely cited and became a memetic anchor point for EHR adoption at the dawn of HITECH. However, its calculation relies on specific treatment of the data; alternative approaches may have led to a different sense of US hospitals' EHR adoption and different subsequent public policy. MATERIALS AND METHODS: We reanalyzed the 2008 American Heart Association Information Technology supplement and complementary sources to produce a range of estimates of EHR adoption. Estimates included the mean and median number of EHR functionalities adopted, figures derived from an item response theory-based approach, and alternative estimates from the published literature. We then plotted an alternative definition of national progress toward hospital EHR adoption from 2008 to 2018. RESULTS: By 2008, 73% of hospitals had begun the transition to an EHR, and the majority of hospitals had adopted at least 6 of the 10 functionalities of a basic system. In the aggregate, national progress toward basic EHR adoption was 58% complete, and, when accounting for measurement error, we estimate that 30% of hospitals may have adopted a basic EHR. DISCUSSION: The approach used to develop the 9% figure resulted in an estimate at the extreme lower bound of what could be derived from the available data and likely did not reflect hospitals' overall progress in EHR adoption. CONCLUSION: The memetic 9% figure shaped nationwide thinking and policy making about EHR adoption; alternative representations of the data may have led to different policy.

Understanding the roles of three academic communities in a prospective learning health ecosystem for diagnostic excellence.

Inaccurate, untimely, and miscommunicated medical diagnoses represent a wicked problem requiring comprehensive and coordinated approaches, such as those demonstrated in the characteristics of learning health systems (LHSs). To appreciate a vision for how LHS methods can optimize processes and outcomes in medical diagnosis (diagnostic excellence), we interviewed 32 individuals with relevant expertise: 18 who have studied diagnostic processes using traditional behavioral science and health services research methods, six focused on machine learning (ML) and artificial intelligence (AI) approaches, and eight multidisciplinary researchers experienced in advocating for and incorporating LHS methods, ie, scalable continuous learning in health care. We report on barriers and facilitators, identified by these subjects, to applying their methods toward optimizing medical diagnosis. We then employ their insights to envision the emergence of a learning ecosystem that leverages the tools of each of the three research groups to advance diagnostic excellence. We found that these communities represent a natural fit forward, in which together, they can better measure diagnostic processes and close the loop of putting insights into practice. Members of the three academic communities will need to network and bring in additional stakeholders before they can design and implement the necessary infrastructure that would support ongoing learning of diagnostic processes at an economy of scale and scope.

A "fundamental theorem" of biomedical informatics.

This paper proposes, in words and pictures, a "fundamental theorem" to help clarify what informatics is and what it is not. In words, the theorem stipulates that a person working in partnership with an information resource is "better" than that same person unassisted. The theorem is applicable to health care, research, education, and administrative activities. Three corollaries to the theorem illustrate that informatics is more about people than technology; that in order for the theorem to hold, resources must be informative in addition to being correct; and that the theorem can fail to hold for reasons explained by understanding the interaction between the person and the resource.

Preparing healthcare delivery organizations for managing computable knowledge.

INTRODUCTION: The growth of data science has led to an explosion in new knowledge alongside various approaches to representing and sharing biomedical knowledge in computable form. These changes have not been matched by an understanding of what healthcare delivery organizations need to do to adapt and continuously deploy computable knowledge. It is therefore important to begin to conceptualize such changes in order to facilitate routine and systematic application of knowledge that improves the health of individuals and populations. METHODS: An AHRQ-funded conference convened a group of experts from a range of fields to analyze the current state of knowledge management in healthcare delivery organizations and describe how it needs to evolve to enable computable knowledge management. Presentations and discussions were recorded and analyzed by the author team to identify foundational concepts and new domains of healthcare delivery organization knowledge management capabilities. RESULTS: Three foundational concepts include 1) the current state of knowledge management in healthcare delivery organizations relies on an outdated biomedical library model, and only a small number of organizations have developed enterprise-scale knowledge management approaches that "push" knowledge in computable form to frontline decisions, 2) the concept of Learning Health Systems creates an imperative for scalable computable knowledge management approaches, and 3) the ability to represent data science discoveries in computable form that is FAIR (findable, accessible, interoperable, reusable) is fundamental to spread knowledge at scale. For healthcare delivery organizations to engage with computable knowledge management at scale, they will need new organizational capabilities across three domains: policies and processes, technology, and people. Examples of specific capabilities were developed. CONCLUSIONS: Healthcare delivery organizations need to substantially scale up and retool their knowledge management approaches in order to benefit from computable biomedical knowledge.

A review of measurement practice in studies of clinical decision support systems 1998-2017.

OBJECTIVE: To assess measurement practice in clinical decision support evaluation studies. MATERIALS AND METHODS: We identified empirical studies evaluating clinical decision support systems published from 1998 to 2017. We reviewed titles, abstracts, and full paper contents for evidence of attention to measurement validity, reliability, or reuse. We used Friedman and Wyatt's typology to categorize the studies. RESULTS: There were 391 studies that met the inclusion criteria. Study types in this cohort were primarily field user effect studies (n = 210) or problem impact studies (n = 150). Of those, 280 studies (72%) had no evidence of attention to measurement methodology, and 111 (28%) had some evidence with 33 (8%) offering validity evidence; 45 (12%) offering reliability evidence; and 61 (16%) reporting measurement artefact reuse. DISCUSSION: Only 5 studies offered validity assessment within the study. Valid measures were predominantly observed in problem impact studies with the majority of measures being clinical or patient reported outcomes with validity measured elsewhere. CONCLUSION: Measurement methodology is frequently ignored in empirical studies of clinical decision support systems and particularly so in field user effect studies. Authors may in fact be attending to measurement considerations and not reporting this or employing methods of unknown validity and reliability in their studies. In the latter case, reported study results may be biased and effect sizes misleading. We argue that replication studies to strengthen the evidence base require greater attention to measurement practice in health informatics research.