Variable Impact of Medical Scribes on Physician Electronic Health Record Documentation Practices: A Quantitative Analysis Across a Large, Integrated Health-System.

BACKGROUND: Medical scribes have been utilized to reduce electronic health record (EHR) associated documentation burden. Although evidence suggests benefits to scribes, no large-scale studies have quantitatively evaluated scribe impact on physician documentation across clinical settings. This study aimed to evaluate the effect of scribes on physician EHR documentation behaviors and performance. METHODS: This retrospective cohort study used EHR audit log data from a large academic health system to evaluate clinical documentation for all ambulatory encounters between January 2014 and December 2019 to evaluate the effect of scribes on physician documentation behaviors. Scribe services were provided on a first-come, first-served basis on physician request. Based on a physician's scribe use, encounters were grouped into 3 categories: never using a scribe, prescribe (before scribe use), or using a scribe. Outcomes included chart closure time, the proportion of delinquent charts, and charts closed after-hours. RESULTS: Three hundred ninety-five physicians (23% scribe users) across 29 medical subspecialties, encompassing 1,132,487 encounters, were included in the analysis. At baseline, scribe users had higher chart closure time, delinquent charts, and after-hours documentation than physicians who never used scribes. Among scribe users, the difference in outcome measures postscribe compared with baseline varied, and using a scribe rarely resulted in outcome measures approaching a range similar to the performance levels of nonusing physicians. In addition, there was variability in outcome measures across medical specialties and within similar subspecialties. CONCLUSION: Although scribes may improve documentation efficiency among some physicians, not all will improve EHR-related documentation practices. Different strategies may help to optimize documentation behaviors of physician-scribe dyads and maximize outcomes of scribe implementation.

Prediction of multiclass surgical outcomes in glaucoma using multimodal deep learning based on free-text operative notes and structured EHR data.

OBJECTIVE: Surgical outcome prediction is challenging but necessary for postoperative management. Current machine learning models utilize pre- and post-op data, excluding intraoperative information in surgical notes. Current models also usually predict binary outcomes even when surgeries have multiple outcomes that require different postoperative management. This study addresses these gaps by incorporating intraoperative information into multimodal models for multiclass glaucoma surgery outcome prediction. MATERIALS AND METHODS: We developed and evaluated multimodal deep learning models for multiclass glaucoma trabeculectomy surgery outcomes using both structured EHR data and free-text operative notes. We compare those to baseline models that use structured EHR data exclusively, or neural network models that leverage only operative notes. RESULTS: The multimodal neural network had the highest performance with a macro AUROC of 0.750 and F1 score of 0.583. It outperformed the baseline machine learning model with structured EHR data alone (macro AUROC of 0.712 and F1 score of 0.486). Additionally, the multimodal model achieved the highest recall (0.692) for hypotony surgical failure, while the surgical success group had the highest precision (0.884) and F1 score (0.775). DISCUSSION: This study shows that operative notes are an important source of predictive information. The multimodal predictive model combining perioperative notes and structured pre- and post-op EHR data outperformed other models. Multiclass surgical outcome prediction can provide valuable insights for clinical decision-making. CONCLUSIONS: Our results show the potential of deep learning models to enhance clinical decision-making for postoperative management. They can be applied to other specialties to improve surgical outcome predictions.

Healthcare utilization is a collider: an introduction to collider bias in EHR data reuse.

OBJECTIVES: Collider bias is a common threat to internal validity in clinical research but is rarely mentioned in informatics education or literature. Conditioning on a collider, which is a variable that is the shared causal descendant of an exposure and outcome, may result in spurious associations between the exposure and outcome. Our objective is to introduce readers to collider bias and its corollaries in the retrospective analysis of electronic health record (EHR) data. TARGET AUDIENCE: Collider bias is likely to arise in the reuse of EHR data, due to data-generating mechanisms and the nature of healthcare access and utilization in the United States. Therefore, this tutorial is aimed at informaticians and other EHR data consumers without a background in epidemiological methods or causal inference. SCOPE: We focus specifically on problems that may arise from conditioning on forms of healthcare utilization, a common collider that is an implicit selection criterion when one reuses EHR data. Directed acyclic graphs (DAGs) are introduced as a tool for identifying potential sources of bias during study design and planning. References for additional resources on causal inference and DAG construction are provided.

Medical Record Closure Practices of Physicians Before and After the Use of Medical Scribes.

This study uses electronic health record data to evaluate medical record closure outcomes before and after the use of medical scribes at a large academic medical center.

Issues With Variability in Electronic Health Record Data About Race and Ethnicity: Descriptive Analysis of the National COVID Cohort Collaborative Data Enclave.

BACKGROUND: The adverse impact of COVID-19 on marginalized and under-resourced communities of color has highlighted the need for accurate, comprehensive race and ethnicity data. However, a significant technical challenge related to integrating race and ethnicity data in large, consolidated databases is the lack of consistency in how data about race and ethnicity are collected and structured by health care organizations. OBJECTIVE: This study aims to evaluate and describe variations in how health care systems collect and report information about the race and ethnicity of their patients and to assess how well these data are integrated when aggregated into a large clinical database. METHODS: At the time of our analysis, the National COVID Cohort Collaborative (N3C) Data Enclave contained records from 6.5 million patients contributed by 56 health care institutions. We quantified the variability in the harmonized race and ethnicity data in the N3C Data Enclave by analyzing the conformance to health care standards for such data. We conducted a descriptive analysis by comparing the harmonized data available for research purposes in the database to the original source data contributed by health care institutions. To make the comparison, we tabulated the original source codes, enumerating how many patients had been reported with each encoded value and how many distinct ways each category was reported. The nonconforming data were also cross tabulated by 3 factors: patient ethnicity, the number of data partners using each code, and which data models utilized those particular encodings. For the nonconforming data, we used an inductive approach to sort the source encodings into categories. For example, values such as "Declined" were grouped with "Refused," and "Multiple Race" was grouped with "Two or more races" and "Multiracial." RESULTS: "No matching concept" was the second largest harmonized concept used by the N3C to describe the race of patients in their database. In addition, 20.7% of the race data did not conform to the standard; the largest category was data that were missing. Hispanic or Latino patients were overrepresented in the nonconforming racial data, and data from American Indian or Alaska Native patients were obscured. Although only a small proportion of the source data had not been mapped to the correct concepts (0.6%), Black or African American and Hispanic/Latino patients were overrepresented in this category. CONCLUSIONS: Differences in how race and ethnicity data are conceptualized and encoded by health care institutions can affect the quality of the data in aggregated clinical databases. The impact of data quality issues in the N3C Data Enclave was not equal across all races and ethnicities, which has the potential to introduce bias in analyses and conclusions drawn from these data. Transparency about how data have been transformed can help users make accurate analyses and inferences and eventually better guide clinical care and public policy.

The Role of Informatics in Implementing Guidelines for Chronic Opioid Therapy Risk Assessment in Primary Care: A Narrative Review Informed by the Socio-Technical Model.

Approximately 2 million Americans live with opioid use disorder (OUD), most of whom also have chronic pain. The economic burden of chronic pain and prescription opioid misuse runs into billions of dollars. Patients on prescription opioids for chronic non-cancer pain (CNCP) are at increased risk for OUD and overdose. By adhering to the Center for Disease Control and Prevention (CDC) opioid prescribing guidelines, primary care providers (PCPs) have the potential to improve patient outcomes. But numerous provider, patient, and practice-specific factors challenge adherence to guidelines in primary care. Many of the barriers may be mediated by informatics interventions, but gaps in knowledge and unmet needs exist. This narrative review examines the risk assessment and harm reduction process in a socio-technical context to highlight the gaps in knowledge and unmet needs that can be mediated through informatics intervention.

Social Determinants of Health Factors for Gene-Environment COVID-19 Research: Challenges and Opportunities.

The characteristics of a person's health status are often guided by how they live, grow, learn, their genetics, as well as their access to health care. Yet, all too often, studies examining the relationship between social determinants of health (behavioral, sociocultural, and physical environmental factors), the role of demographics, and health outcomes poorly represent these relationships, leading to misinterpretations, limited study reproducibility, and datasets with limited representativeness and secondary research use capacity. This is a profound hurdle in what questions can or cannot be rigorously studied about COVID-19. In practice, gene-environment interactions studies have paved the way for including these factors into research. Similarly, our understanding of social determinants of health continues to expand with diverse data collection modalities as health systems, patients, and community health engagement aim to fill the knowledge gaps toward promoting health and wellness. Here, a conceptual framework is proposed, adapted from the population health framework, socioecological model, and causal modeling in gene-environment interaction studies to integrate the core constructs from each domain with practical considerations needed for multidisciplinary science.

Unraveling COVID-19: A Large-Scale Characterization of 4.5 Million COVID-19 Cases Using CHARYBDIS.

Purpose: Routinely collected real world data (RWD) have great utility in aiding the novel coronavirus disease (COVID-19) pandemic response. Here we present the international Observational Health Data Sciences and Informatics (OHDSI) Characterizing Health Associated Risks and Your Baseline Disease In SARS-COV-2 (CHARYBDIS) framework for standardisation and analysis of COVID-19 RWD. Patients and Methods: We conducted a descriptive retrospective database study using a federated network of data partners in the United States, Europe (the Netherlands, Spain, the UK, Germany, France and Italy) and Asia (South Korea and China). The study protocol and analytical package were released on 11th June 2020 and are iteratively updated via GitHub. We identified three non-mutually exclusive cohorts of 4,537,153 individuals with a clinical COVID-19 diagnosis or positive test, 886,193 hospitalized with COVID-19, and 113,627 hospitalized with COVID-19 requiring intensive services. Results: We aggregated over 22,000 unique characteristics describing patients with COVID-19. All comorbidities, symptoms, medications, and outcomes are described by cohort in aggregate counts and are readily available online. Globally, we observed similarities in the USA and Europe: more women diagnosed than men but more men hospitalized than women, most diagnosed cases between 25 and 60 years of age versus most hospitalized cases between 60 and 80 years of age. South Korea differed with more women than men hospitalized. Common comorbidities included type 2 diabetes, hypertension, chronic kidney disease and heart disease. Common presenting symptoms were dyspnea, cough and fever. Symptom data availability was more common in hospitalized cohorts than diagnosed. Conclusion: We constructed a global, multi-centre view to describe trends in COVID-19 progression, management and evolution over time. By characterising baseline variability in patients and geography, our work provides critical context that may otherwise be misconstrued as data quality issues. This is important as we perform studies on adverse events of special interest in COVID-19 vaccine surveillance.

The quality of social determinants data in the electronic health record: a systematic review.

OBJECTIVE: The aim of this study was to collect and synthesize evidence regarding data quality problems encountered when working with variables related to social determinants of health (SDoH). MATERIALS AND METHODS: We conducted a systematic review of the literature on social determinants research and data quality and then iteratively identified themes in the literature using a content analysis process. RESULTS: The most commonly represented quality issue associated with SDoH data is plausibility (n = 31, 41%). Factors related to race and ethnicity have the largest body of literature (n = 40, 53%). The first theme, noted in 62% (n = 47) of articles, is that bias or validity issues often result from data quality problems. The most frequently identified validity issue is misclassification bias (n = 23, 30%). The second theme is that many of the articles suggest methods for mitigating the issues resulting from poor social determinants data quality. We grouped these into 5 suggestions: avoid complete case analysis, impute data, rely on multiple sources, use validated software tools, and select addresses thoughtfully. DISCUSSION: The type of data quality problem varies depending on the variable, and each problem is associated with particular forms of analytical error. Problems encountered with the quality of SDoH data are rarely distributed randomly. Data from Hispanic patients are more prone to issues with plausibility and misclassification than data from other racial/ethnic groups. CONCLUSION: Consideration of data quality and evidence-based quality improvement methods may help prevent bias and improve the validity of research conducted with SDoH data.

An Analysis of Two Sources of Cardiology Patient Data to Measure Medication Agreement.

Errors and incompleteness in electronic health record (EHR) medication lists can result in medical errors. To reduce errors in these medication lists, clinicians use patient self-reported data to reconcile EHR data. We assessed the agreement between patient self-reported medications and medications recorded in the EHR for six medication classes related to cardiovascular care and used logistic regression models to determine which patient-related factors were associated with the disagreement between these two information sources. From our 297 patients, we found self-reported medications had an overall above-average agreement with the EHR (? = .727). We observed the highest agreement level for statins (? = .831) and the lowest for other antihypertensives (? = .465). Agreement was less likely for Hispanic and male patients. We also performed an in-depth error analysis of different types of disagreement beyond medication names, which revealed that the most frequent type of disagreement was mismatched dosages.

Extracting Patient-level Social Determinants of Health into the OMOP Common Data Model.

Deficiencies in data sharing capabilities limit Social Determinants of Health (SDoH) analysis as part of COVID-19 research. The National COVID Cohort Collaborative (N3C) is an example of an Electronic Health Record (EHR) database of patients tested for COVID-19 that could benefit from a SDoH elements framework that captures various screening instruments in EHR data warehouse systems. This paper uses the University of Washington Enterprise Data Warehouse (a data contributor to N3C) to demonstrate how SDoH can be represented and managed to be made available within an OMOP common data model. We found that these data varied by type of social determinants data and where it was collected, in the time period that it was collected, and in how it was represented.

Chart Completion Time of Attending Physicians While Using Medical Scribes.

Medical scribes have become a widely used strategy to optimize how providers document in the electronic health record. To date, literature regarding the impact of scribes on time to complete documentation is limited. We conducted a retrospective, descriptive study of chart completion time among providers using scribes at our organization. A total of 148,410 scribed encounters, across 55 different clinics, were analyzed to determine variations in chart completion time. There was a significant variance in completion time between specialty groups and clinics within each specialty. Additionally, chart completion time was highly variable between providers working in the same clinic. These patterns were observed across all specialties included in our analysis. Our results suggest a higher level of variability with respect to chart completion when utilizing scribes than previously anticipated.

Charting the life course: Emerging opportunities to advance scientific approaches using life course research.

Life course research embraces the complexity of health and disease development, tackling the extensive interactions between genetics and environment. This interdisciplinary blueprint, or theoretical framework, offers a structure for research ideas and specifies relationships between related factors. Traditionally, methodological approaches attempt to reduce the complexity of these dynamic interactions and decompose health into component parts, ignoring the complex reciprocal interaction of factors that shape health over time. New methods that match the epistemological foundation of the life course framework are needed to fully explore adaptive, multilevel, and reciprocal interactions between individuals and their environment. The focus of this article is to (1) delineate the differences between lifespan and life course research, (2) articulate the importance of complex systems science as a methodological framework in the life course research toolbox to guide our research questions, (3) raise key questions that can be asked within the clinical and translational science domain utilizing this framework, and (4) provide recommendations for life course research implementation, charting the way forward. Recent advances in computational analytics, computer science, and data collection could be used to approximate, measure, and analyze the intertwining and dynamic nature of genetic and environmental factors involved in health development.

Towards augmenting structured EHR data: a comparison of manual chart review and patient self-report.

Structured electronic health record (EHR) data are often used for quality measurement and improvement, clinical research, and other secondary uses. These data, however, are known to suffer from quality problems. There may be value in augmenting structured EHR data to improve data quality, thereby improving the reliability and validity of the conclusions drawn from those data. Focusing on five diagnoses related to cardiovascular care, this paper considers the added value of two alternative data sources: manual chart abstraction and patient self-report. We assess the overall agreement between structured EHR problem list data, abstracted EHR data, and patient self- report; and explore possible causes of disagreement between those sources. Our findings suggest that both chart abstraction and patient self-report contain significantly more diagnoses than the problem list, but that the information they capture is different. Methods for collecting and validating self-reported medical data require further consideration and exploration.

Comparison of EHR-based diagnosis documentation locations to a gold standard for risk stratification in patients with multiple chronic conditions.

OBJECTIVE: To measure variation among four different Electronic Health Record (EHR) system documentation locations versus 'gold standard' manual chart review for risk stratification in patients with multiple chronic illnesses. METHODS: Adults seen in primary care with EHR evidence of at least one of 13 conditions were included. EHRs were manually reviewed to determine presence of active diagnoses, and risk scores were calculated using three different methodologies and five EHR documentation locations. Claims data were used to assess cost and utilization for the following year. Descriptive and diagnostic statistics were calculated for each EHR location. Criterion validity testing compared the gold standard verified diagnoses versus other EHR locations and risk scores in predicting future cost and utilization. RESULTS: Nine hundred patients had 2,179 probable diagnoses. About 70% of the diagnoses from the EHR were verified by gold standard. For a subset of patients having baseline and prediction year data (n=750), modeling showed that the gold standard was the best predictor of outcomes on average for a subset of patients that had these data. However, combining all data sources together had nearly equivalent performance for prediction as the gold standard. CONCLUSIONS: EHR data locations were inaccurate 30% of the time, leading to improvement in overall modeling from a gold standard from chart review for individual diagnoses. However, the impact on identification of the highest risk patients was minor, and combining data from different EHR locations was equivalent to gold standard performance. The reviewer's ability to identify a diagnosis as correct was influenced by a variety of factors, including completeness, temporality, and perceived accuracy of chart data.

Specifications of Clinical Quality Measures and Value Set Vocabularies Shift Over Time: A Study of Change through Implementation Differences.

Clinical quality measures (CQMs) aim to identify gaps in care and to promote evidence-based guidelines. Official CQM definitions consist of a measure's logic and grouped, standardized codes to define key concepts. In this study, we used the official CQM update process to understand how CQMs' meanings change over time. First, we identified differences between the narrative description, logic, and the vocabulary specifications offour standardized CQMs' definitions in subsequent versions (2015, 2016, and 2017). Next, we implemented the various versions in a quality measure calculation registry to understand how the differences affected calculated prevalence of risk and measure performance. Global performance rates changed up to 5.32%, and an increase of up to 28% new patients was observed for key conditions between versions. Updates to definitions that change a measure's logic and choices to include/exclude codes in value set vocabularies changes measurement of quality and likely introduces variation by implementation.

A Data Quality Assessment Guideline for Electronic Health Record Data Reuse.

INTRODUCTION: We describe the formulation, development, and initial expert review of 3x3 Data Quality Assessment (DQA), a dynamic, evidence-based guideline to enable electronic health record (EHR) data quality assessment and reporting for clinical research. METHODS: 3x3 DQA was developed through the triangulation results from three studies: a review of the literature on EHR data quality assessment, a quantitative study of EHR data completeness, and a set of interviews with clinical researchers. Following initial development, the guideline was reviewed by a panel of EHR data quality experts. RESULTS: The guideline embraces the task-dependent nature of data quality and data quality assessment. The core framework includes three constructs of data quality: complete, correct, and current data. These constructs are operationalized according to the three primary dimensions of EHR data: patients, variables, and time. Each of the nine operationalized constructs maps to a methodological recommendation for EHR data quality assessment. The initial expert response to the framework was positive, but improvements are required. DISCUSSION: The initial version of 3x3 DQA promises to enable explicit guideline-based best practices for EHR data quality assessment and reporting. Future work will focus on increasing clarity on how and when 3x3 DQA should be used during the research process, improving the feasibility and ease-of-use of recommendation execution, and clarifying the process for users to determine which operationalized constructs and recommendations are relevant for a given dataset and study.

From Concepts and Codes to Healthcare Quality Measurement: Understanding Variations in Value Set Vocabularies for a Statin Therapy Clinical Quality Measure.

OBJECTIVE: To understand the impact of distinct concept to value set mapping on the measurement of quality of care. BACKGROUND: Clinical quality measures (CQMs) intend to measure the quality of healthcare services provided, and to help promote evidence-based therapies. Most CQMs consist of grouped codes from vocabularies - or 'value sets' - that represent the unique identifiers (i.e., object identifiers), concepts (i.e., value set names), and concept definitions (i.e., code groups) that define a measure's specifications. In the development of a statin therapy CQM, two unique value sets were created by independent measure developers for the same global concepts. METHODS: We first identified differences between the two value set specifications of the same CQM. We then implemented the various versions in a quality measure calculation registry to understand how the differences affected calculated prevalence of risk and measure performance. RESULTS: Global performance rates only differed by 0.8%, but there were up to 2.3 times as many patients included with key conditions, and differing performance rates of 7.5% for patients with 'myocardial infarction' and 3.5% for those with 'ischemic vascular disease'. CONCLUSION: The decisions CQM developers make about which concepts and code groups to include or exclude in value set vocabularies can lead to inaccuracies in the measurement of quality of care. One solution is that developers could provide rationale for these decisions. Endorsements are needed to encourage system vendors, payers, informaticians, and clinicians to collaborate in the creation of more integrated terminology sets.

A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data.

OBJECTIVE: Harmonized data quality (DQ) assessment terms, methods, and reporting practices can establish a common understanding of the strengths and limitations of electronic health record (EHR) data for operational analytics, quality improvement, and research. Existing published DQ terms were harmonized to a comprehensive unified terminology with definitions and examples and organized into a conceptual framework to support a common approach to defining whether EHR data is 'fit' for specific uses. MATERIALS AND METHODS: DQ publications, informatics and analytics experts, managers of established DQ programs, and operational manuals from several mature EHR-based research networks were reviewed to identify potential DQ terms and categories. Two face-to-face stakeholder meetings were used to vet an initial set of DQ terms and definitions that were grouped into an overall conceptual framework. Feedback received from data producers and users was used to construct a draft set of harmonized DQ terms and categories. Multiple rounds of iterative refinement resulted in a set of terms and organizing framework consisting of DQ categories, subcategories, terms, definitions, and examples. The harmonized terminology and logical framework's inclusiveness was evaluated against ten published DQ terminologies. RESULTS: Existing DQ terms were harmonized and organized into a framework by defining three DQ categories: (1) Conformance (2) Completeness and (3) Plausibility and two DQ assessment contexts: (1) Verification and (2) Validation. Conformance and Plausibility categories were further divided into subcategories. Each category and subcategory was defined with respect to whether the data may be verified with organizational data, or validated against an accepted gold standard, depending on proposed context and uses. The coverage of the harmonized DQ terminology was validated by successfully aligning to multiple published DQ terminologies. DISCUSSION: Existing DQ concepts, community input, and expert review informed the development of a distinct set of terms, organized into categories and subcategories. The resulting DQ terms successfully encompassed a wide range of disparate DQ terminologies. Operational definitions were developed to provide guidance for implementing DQ assessment procedures. The resulting structure is an inclusive DQ framework for standardizing DQ assessment and reporting. While our analysis focused on the DQ issues often found in EHR data, the new terminology may be applicable to a wide range of electronic health data such as administrative, research, and patient-reported data. CONCLUSION: A consistent, common DQ terminology, organized into a logical framework, is an initial step in enabling data owners and users, patients, and policy makers to evaluate and communicate data quality findings in a well-defined manner with a shared vocabulary. Future work will leverage the framework and terminology to develop reusable data quality assessment and reporting methods.