Integrating clinical research into electronic health record workflows to support a learning health system.

Objective: Integrating clinical research into routine clinical care workflows within electronic health record systems (EHRs) can be challenging, expensive, and labor-intensive. This case study presents a large-scale clinical research project conducted entirely within a commercial EHR during the COVID-19 pandemic. Case Report: The UCSD and UCSDH COVID-19 NeutraliZing Antibody Project (ZAP) aimed to evaluate antibody levels to SARS-CoV-2 virus in a large population at an academic medical center and examine the association between antibody levels and subsequent infection diagnosis. Results: The project rapidly and successfully enrolled and consented over 2000 participants, integrating the research trial with standing COVID-19 testing operations, staff, lab, and mobile applications. EHR-integration increased enrollment, ease of scheduling, survey distribution, and return of research results at a low cost by utilizing existing resources. Conclusion: The case study highlights the potential benefits of EHR-integrated clinical research, expanding their reach across multiple health systems and facilitating rapid learning during a global health crisis.

Using Large Language Models to Support Content Analysis: A Case Study of ChatGPT for Adverse Event Detection.

This study explores the potential of using large language models to assist content analysis by conducting a case study to identify adverse events (AEs) in social media posts. The case study compares ChatGPT's performance with human annotators' in detecting AEs associated with delta-8-tetrahydrocannabinol, a cannabis-derived product. Using the identical instructions given to human annotators, ChatGPT closely approximated human results, with a high degree of agreement noted: 94.4% (9436/10,000) for any AE detection (Fleiss κ=0.95) and 99.3% (9931/10,000) for serious AEs (κ=0.96). These findings suggest that ChatGPT has the potential to replicate human annotation accurately and efficiently. The study recognizes possible limitations, including concerns about the generalizability due to ChatGPT's training data, and prompts further research with different models, data sources, and content analysis tasks. The study highlights the promise of large language models for enhancing the efficiency of biomedical research.

Consequences of a Health System Not Knowing Which Patients Are Deceased.

This quality improvement study in a California health system investigates the proportion of active patients who were deceased but not noted as such in the electronic health record (EHR), as well as encounters after death.

Development & Deployment of a Real-time Healthcare Predictive Analytics Platform.

The deployment of predictive analytic algorithms that can safely and seamlessly integrate into existing healthcare workflows remains a significant challenge. Here, we present a scalable, cloud-based, fault-tolerant platform that is capable of extracting and processing electronic health record (EHR) data for any patient at any time following admission and transferring results back into the EHR. This platform has been successfully deployed within the UC San Diego Health system and utilizes interoperable data standards to enable portability.Clinical relevance- This platform is currently hosting a deep learning model for the early prediction of sepsis that is operational in two emergency departments.

Clinical Characteristics and Outcomes of Drug-Induced Acute Kidney Injury Cases.

Introduction: Drug-induced acute kidney injury (DI-AKI) is a frequent adverse event. The identification of DI-AKI is challenged by competing etiologies, clinical heterogeneity among patients, and a lack of accurate diagnostic tools. Our research aims to describe the clinical characteristics and predictive variables of DI-AKI. Methods: We analyzed data from the Drug-Induced Renal Injury Consortium (DIRECT) study (NCT02159209), an international, multicenter, observational cohort study of enriched clinically adjudicated DI-AKI cases. Cases met the primary inclusion criteria if the patient was exposed to at least 1 nephrotoxic drug for a minimum of 24 hours prior to AKI onset. Cases were clinically adjudicated, and inter-rater reliability (IRR) was measured using Krippendorff's alpha. Variables associated with DI-AKI were identified using L1 regularized multivariable logistic regression. Model performance was assessed using the area under the receiver operating characteristic curve (ROC AUC). Results: A total of 314 AKI cases met the eligibility criteria for this analysis, and 271 (86%) cases were adjudicated as DI-AKI. The majority of the AKI cases were recruited from the United States (68%). The most frequent causal nephrotoxic drugs were vancomycin (48.7%), nonsteroidal antiinflammatory drugs (18.2%), and piperacillin/tazobactam (17.8%). The IRR for DI-AKI adjudication was 0.309. The multivariable model identified age, vascular capacity, hyperglycemia, infections, pyuria, serum creatinine (SCr) trends, and contrast media as significant predictors of DI-AKI with good performance (ROC AUC 0.86). Conclusion: The identification of DI-AKI is challenging even with comprehensive adjudication by experienced nephrologists. Our analysis identified key clinical characteristics and outcomes of DI-AKI compared to other AKI etiologies.

Ascertaining Whether an Intelligent Voice Assistant Can Meet Older Adults' Health-Related Needs in the Context of a Geriatrics 5Ms Framework.

The Geriatrics 5Ms: Medications, Mind, Mobility, what Matters most and Multicomplexity is a framework to address the complex needs of older adults. Intelligent Voice Assistants (IVAs) are increasingly popular and have potential to support health-related needs of older adults. We utilized previously collected qualitative data on older adults' views of how an IVA may address their health-related needs and ascertained their fit into the Geriatrics 5Ms framework. The codes describing health challenges and potential IVA solutions fit the framework: (1) Medications: difficulty remembering medications. SOLUTION: reminders. (2) Mind: isolation, anxiety, memory loss. SOLUTION: companionship, memory aids. (3) Mobility: barriers to exercise. SOLUTION: incentives, exercise ideas. (4) Matters most: eating healthy foods. SOLUTION: suggest and order nutritious foods, (5) Multicomplexity; managing multimorbidity. SOLUTION: symptom tracking and communicating with health care professionals. Incorporating the 5Ms framework into IVA design can aid in addressing health care priorities of older adults.

Early Ascertainment of Breast Cancer Diagnoses Comparing Self-Reported Questionnaires and Electronic Health Record Data Warehouse: The WISDOM Study.

PURPOSE: The goal of this study was to use real-world data sources that may be faster and more complete than self-reported data alone, and timelier than cancer registries, to ascertain breast cancer cases in the ongoing screening trial, the WISDOM Study. METHODS: We developed a data warehouse procedural process (DWPP) to identify breast cancer cases from a subgroup of WISDOM participants (n = 11,314) who received breast-related care from a University of California Health Center in the period 2012-2021 by searching electronic health records (EHRs) in the University of California Data Warehouse (UCDW). Incident breast cancer diagnoses identified by the DWPP were compared with those identified by self-report via annual follow-up online questionnaires. RESULTS: Our study identified 172 participants with confirmed breast cancer diagnoses in the period 2016-2021 by the following sources: 129 (75%) by both self-report and DWPP, 23 (13%) by DWPP alone, and 20 (12%) by self-report only. Among those with International Classification of Diseases 10th revision cancer diagnostic codes, no diagnosis was confirmed in 18% of participants. CONCLUSION: For diagnoses that occurred ≥20 months before the January 1, 2022, UCDW data pull, WISDOM self-reported data via annual questionnaire achieved high accuracy (96%), as confirmed by the cancer registry. More rapid cancer ascertainment can be achieved by combining self-reported data with EHR data from a health system data warehouse registry, particularly to address self-reported questionnaire issues such as timing delays (ie, time lag between participant diagnoses and the submission of their self-reported questionnaire typically ranges from a month to a year) and lack of response. Although cancer registry reporting often is not as timely, it does not require verification as does the DWPP or self-report from annual questionnaires.

Concept Coverage Analysis of Ophthalmic Infections and Trauma among the Standardized Medical Terminologies SNOMED-CT, ICD-10-CM, and ICD-11.

Purpose: Widespread electronic health record adoption has generated a large volume of data and emphasized the need for standardized terminology to describe clinical concepts. Here, we undertook a systematic concept coverage analysis to determine the representation of clinical concepts in ophthalmic infection and ophthalmic trauma among standardized medical terminologies, including the Systematized Nomenclature of Medicine Clinical Terms (SNOMED-CT), the International Classification of Diseases (ICD) version 10 with clinical modifications (ICD-10-CM), and ICD version 11 (ICD-11). Design: Extraction of concepts related to ophthalmic infection and ophthalmic trauma and structured search in terminology browsers. Data Sources: The American Academy of Ophthalmology Basic and Clinical Science Course (BCSC), SNOMED-CT, and ICD-10-CM terminologies from the Observational Health Data Sciences and Informatics Athena browser, and the ICD-11 terminology browser. Methods: Concepts pertaining to ophthalmic infection and ophthalmic trauma were extracted from the 2022 BCSC free text and index terms. We searched terminology browsers to identify corresponding codes and classified the extent of semantic alignment as equal, wide, narrow, or unmatched in each terminology. The overlap of equal concepts in each terminology was represented in a Venn diagram. Main Outcome Measures: Proportions of clinical concepts with corresponding codes at various levels of semantic alignment. Results: A total of 443 concepts were identified: 304 concepts related to ophthalmic infection and 139 concepts related to ophthalmic trauma. The SNOMED-CT had the highest proportion of equal coverage, with 82.0% (249 of 304) among concepts related to ophthalmic infection and 82.0% (115 of 139) among concepts related to ophthalmic trauma. Across all concepts, 28% (124 of 443) were classified as equal in ICD-10-CM and 52.8% (234 of 443) were classified as equal in ICD-11. Conclusions: The SNOMED-CT had significantly better semantic alignment than ICD-10-CM and ICD-11 for ophthalmic infections and ophthalmic trauma. This demonstrates opportunity for continuing advancement of representation of ophthalmic concepts in standardized medical terminologies.

Evaluating Artificial Intelligence Responses to Public Health Questions.

This cross-sectional study analyzes the quality of ChatGPT responses to public health questions.

Clinical characteristics and comorbidities associated with SARS-CoV-2 breakthrough infection in the University of California Healthcare Systems.

BACKGROUND: To evaluate the degree to which clinical comorbidities or combinations of comorbidities are associated with SARS-CoV-2 breakthrough infection. MATERIALS AND METHODS: A breakthrough infection was defined as a positive test at least 14 days after a full vaccination regimen. Logistic regression was used to calculate aORs, which were adjusted for age, sex, and race information. RESULTS: A total of 110,380 patients from the UC CORDS database were included. After adjustment, stage 5 CKD due to hypertension (aOR: 7.33; 95% CI: 4.86-10.69; p<.001; power=1) displayed higher odds of infection than any other comorbidity. Lung transplantation history (aOR: 4.79; 95% CI: 3.25-6.82; p<.001; power= 1), coronary atherosclerosis (aOR: 2.12; 95% CI: 1.77-2.52; p<.001; power=1), and vitamin D deficiency (aOR: 1.87; 95% CI: 1.69-2.06; p<.001; power=1) were significantly correlated to breakthrough infection. Patients with obesity in addition to essential hypertension (aOR: 1.74; 95% CI: 1.51-2.01; p<.001; power=1) and anemia (aOR: 1.80; 95% CI: 1.47-2.19; p<.001; power=1) were at additional risk of breakthrough infection compared to those with essential hypertension and anemia alone. CONCLUSIONS: Further measures should be taken to prevent breakthrough infection for individuals with these conditions, such as acquiring additional doses of the SARS-CoV-2 vaccine to boost immunity.

Development & Deployment of a Real-time Healthcare Predictive Analytics Platform.

The deployment of predictive analytic algorithms that can safely and seamlessly integrate into existing healthcare workflows remains a significant challenge. Here, we present a scalable, cloud-based, fault-tolerant platform that is capable of extracting and processing electronic health record (EHR) data for any patient at any time following admission and transferring results back into the EHR. This platform has been successfully deployed within the UC San Diego Health system and utilizes interoperable data standards to enable portability.

Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum.

Importance: The rapid expansion of virtual health care has caused a surge in patient messages concomitant with more work and burnout among health care professionals. Artificial intelligence (AI) assistants could potentially aid in creating answers to patient questions by drafting responses that could be reviewed by clinicians. Objective: To evaluate the ability of an AI chatbot assistant (ChatGPT), released in November 2022, to provide quality and empathetic responses to patient questions. Design, Setting, and Participants: In this cross-sectional study, a public and nonidentifiable database of questions from a public social media forum (Reddit's r/AskDocs) was used to randomly draw 195 exchanges from October 2022 where a verified physician responded to a public question. Chatbot responses were generated by entering the original question into a fresh session (without prior questions having been asked in the session) on December 22 and 23, 2022. The original question along with anonymized and randomly ordered physician and chatbot responses were evaluated in triplicate by a team of licensed health care professionals. Evaluators chose "which response was better" and judged both "the quality of information provided" (very poor, poor, acceptable, good, or very good) and "the empathy or bedside manner provided" (not empathetic, slightly empathetic, moderately empathetic, empathetic, and very empathetic). Mean outcomes were ordered on a 1 to 5 scale and compared between chatbot and physicians. Results: Of the 195 questions and responses, evaluators preferred chatbot responses to physician responses in 78.6% (95% CI, 75.0%-81.8%) of the 585 evaluations. Mean (IQR) physician responses were significantly shorter than chatbot responses (52 [17-62] words vs 211 [168-245] words; t = 25.4; P < .001). Chatbot responses were rated of significantly higher quality than physician responses (t = 13.3; P < .001). The proportion of responses rated as good or very good quality (≥ 4), for instance, was higher for chatbot than physicians (chatbot: 78.5%, 95% CI, 72.3%-84.1%; physicians: 22.1%, 95% CI, 16.4%-28.2%;). This amounted to 3.6 times higher prevalence of good or very good quality responses for the chatbot. Chatbot responses were also rated significantly more empathetic than physician responses (t = 18.9; P < .001). The proportion of responses rated empathetic or very empathetic (≥4) was higher for chatbot than for physicians (physicians: 4.6%, 95% CI, 2.1%-7.7%; chatbot: 45.1%, 95% CI, 38.5%-51.8%; physicians: 4.6%, 95% CI, 2.1%-7.7%). This amounted to 9.8 times higher prevalence of empathetic or very empathetic responses for the chatbot. Conclusions: In this cross-sectional study, a chatbot generated quality and empathetic responses to patient questions posed in an online forum. Further exploration of this technology is warranted in clinical settings, such as using chatbot to draft responses that physicians could then edit. Randomized trials could assess further if using AI assistants might improve responses, lower clinician burnout, and improve patient outcomes.

AI in Pathology: What could possibly go wrong?

The field of medicine is undergoing rapid digital transformation. Pathologists are now striving to digitize their data, workflows, and interpretations, assisted by the enabling development of whole-slide imaging. Going digital means that the analog process of human diagnosis can be augmented or even replaced by rapidly evolving AI approaches, which are just now entering into clinical practice. But with such progress comes challenges that reflect a variety of stressors, including the impact of unrepresentative training data with accompanying implicit bias, data privacy concerns, and fragility of algorithm performance. Beyond such core digital aspects, considerations arise related to difficulties presented by changing disease presentations, diagnostic approaches, and therapeutic options. While some tools such as data federation can help with broadening data diversity while preserving expertise and local control, they may not be the full answer to some of these issues. The impact of AI in pathology on the field's human practitioners is still very much unknown: installation of unconscious bias and deference to AI guidance need to be understood and addressed. If AI is widely adopted, it may remove many inefficiencies in daily practice and compensate for staff shortages. It may also cause practitioner deskilling, dethrilling, and burnout. We discuss the technological, clinical, legal, and sociological factors that will influence the adoption of AI in pathology, and its eventual impact for good or ill.

Enhancing research data infrastructure to address the opioid epidemic: the Opioid Overdose Network (O2-Net).

Opioid Overdose Network is an effort to generalize and adapt an existing research data network, the Accrual to Clinical Trials (ACT) Network, to support design of trials for survivors of opioid overdoses presenting to emergency departments (ED). Four institutions (Medical University of South Carolina [MUSC], Dartmouth Medical School [DMS], University of Kentucky [UK], and University of California San Diego [UCSD]) worked to adapt the ACT network. The approach that was taken to enhance the ACT network focused on 4 activities: cloning and extending the ACT infrastructure, developing an e-phenotype and corresponding registry, developing portable natural language processing tools to enhance data capture, and developing automated documentation templates to enhance extended data capture. Overall, initial results suggest that tailoring of existing multipurpose federated research networks to specific tasks is feasible; however, substantial efforts are required for coordination of the subnetwork and development of new tools for extension of available data. The initial output of the project was a new approach to decision support for the prescription of naloxone for home use in the ED, which is under further study within the network.

Implementation of Patient Engagement Tools in Electronic Health Records to Enhance Patient-Centered Communication: Protocol for Feasibility Evaluation and Preliminary Results.

BACKGROUND: Patient-physician communication during clinical encounters is essential to ensure quality of care. Many studies have attempted to improve patient-physician communication. Incorporating patient priorities into agenda setting and medical decision-making are fundamental to patient-centered communication. Efficient and scalable approaches are needed to empower patients to speak up and prepare physicians to respond. Leveraging electronic health records (EHRs) in engaging patients and health care teams has the potential to enhance the integration of patient priorities in clinical encounters. A systematic approach to eliciting and documenting patient priorities before encounters could facilitate effective communication in such encounters. OBJECTIVE: In this paper, we report the design and implementation of a set of EHR tools built into clinical workflows for facilitating patient-physician joint agenda setting and the documentation of patient concerns in the EHRs for ambulatory encounters. METHODS: We engaged health information technology leaders and users in three health care systems for developing and implementing a set of EHR tools. The goal of these tools is to standardize the elicitation of patient priorities by using a previsit "patient important issue" questionnaire distributed through the patient portal to the EHR. We built additional EHR documentation tools to facilitate patient-staff communication when the staff records the vital signs and the reason for the visit in the EHR while in the examination room, with a simple transmission method for physicians to incorporate patient concerns in EHR notes. RESULTS: The study is ongoing. The anticipated completion date for survey data collection is November 2021. A total of 34,037 primary care patients from three health systems (n=26,441; n=5136; and n=2460 separately recruited from each system) used the previsit patient important issue questionnaire in 2020. The adoption of the digital previsit questionnaire during the COVID-19 pandemic was much higher in one health care system because it expanded the use of the questionnaire from physicians participating in trials to all primary care providers midway through the year. It also required the use of this previsit questionnaire for eCheck-ins, which are required for telehealth encounters. Physicians and staff suggested anecdotally that this questionnaire helped patient-clinician communication, particularly during the COVID-19 pandemic. CONCLUSIONS: EHR tools have the potential to facilitate the integration of patient priorities into agenda setting and documentation in real-world primary care practices. Early results suggest the feasibility and acceptability of such digital tools in three health systems. EHR tools can support patient engagement and clinicians' work during in-person and telehealth visits. They could potentially exert a sustained influence on patient and clinician communication behaviors in contrast to prior ad hoc educational efforts targeting patients or clinicians. TRIAL REGISTRATION: ClinicalTrials.gov NCT03385512; https://clinicaltrials.gov/ct2/show/NCT03385512. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/30431.

Predicting Mortality in Critical Care Patients with Fungemia Using Structured and Unstructured Data.

Fungemia is a life-threatening infection, but predictive models of in-patient mortality in this infection are few. In this study, we developed models predicting all-cause in-hospital mortality among 265 fungemic patients in the Medical Information Mart for Intensive Care (MIMIC-III) database using both structured and unstructured data. Structured data models included multivariable logistic regression, extreme gradient boosting, and stacked ensemble models. Unstructured data models were developed using Amazon Comprehend Medical and BioWordVec embeddings in logistic regression, convolutional neural networks (CNNs), and recurrent neural networks (RNNs). We evaluated models trained on all notes, notes from only the first three days of hospitalization, and models trained on only physician notes. The best-performing structured data model was a multivariable logistic regression model that achieved an accuracy of 0.74 and AUC of 0.76. Liver disease, acute renal failure, and intubation were some of the top features driving prediction in multiple models. CNNs using unstructured data achieved similar performance even when trained with notes from only the first three days of hospitalization. The best-performing unstructured data models used the Amazon Comprehend Medical document classifier and CNNs, achieving accuracy ranging from 0.99-1.00, and AUCs of 1.00. Therefore, unstructured data - particularly notes composed by physicians - offer added predictive value over models based on structured data alone.

Repurposing Clinical Decision Support System Data to Measure Dosing Errors and Clinician-Level Quality of Care.

We aimed to develop and validate an instrument to detect hospital medication prescribing errors using repurposed clinical decision support system data. Despite significant efforts to eliminate medication prescribing errors, these events remain common in hospitals. Data from clinical decision support systems have not been used to identify prescribing errors as an instrument for physician-level performance. We evaluated medication order alerts generated by a knowledge-based electronic prescribing system occurring in one large academic medical center's acute care facilities for patient encounters between 2009 and 2012. We developed and validated an instrument to detect medication prescribing errors through a clinical expert panel consensus process to assess physician quality of care. Six medication prescribing alert categories were evaluated for inclusion, one of which - dose - was included in the algorithm to detect prescribing errors. The instrument was 93% sensitive (recall), 51% specific, 40% precise, 62% accurate, with an F1 score of 55%, positive predictive value of 96%, and a negative predictive value of 32%. Using repurposed electronic prescribing system data, dose alert overrides can be used to systematically detect medication prescribing errors occurring in an inpatient setting with high sensitivity.

Text Processing for Detection of Fungal Ocular Involvement in Critical Care Patients: Cross-Sectional Study.

BACKGROUND: Fungal ocular involvement can develop in patients with fungal bloodstream infections and can be vision-threatening. Ocular involvement has become less common in the current era of improved antifungal therapies. Retrospectively determining the prevalence of fungal ocular involvement is important for informing clinical guidelines, such as the need for routine ophthalmologic consultations. However, manual retrospective record review to detect cases is time-consuming. OBJECTIVE: This study aimed to determine the prevalence of fungal ocular involvement in a critical care database using both structured and unstructured electronic health record (EHR) data. METHODS: We queried microbiology data from 46,467 critical care patients over 12 years (2000-2012) from the Medical Information Mart for Intensive Care III (MIMIC-III) to identify 265 patients with culture-proven fungemia. For each fungemic patient, demographic data, fungal species present in blood culture, and risk factors for fungemia (eg, presence of indwelling catheters, recent major surgery, diabetes, immunosuppressed status) were ascertained. All structured diagnosis codes and free-text narrative notes associated with each patient's hospitalization were also extracted. Screening for fungal endophthalmitis was performed using two approaches: (1) by querying a wide array of eye- and vision-related diagnosis codes, and (2) by utilizing a custom regular expression pipeline to identify and collate relevant text matches pertaining to fungal ocular involvement. Both approaches were validated using manual record review. The main outcome measure was the documentation of any fungal ocular involvement. RESULTS: In total, 265 patients had culture-proven fungemia, with Candida albicans (n=114, 43%) and Candida glabrata (n=74, 28%) being the most common fungal species in blood culture. The in-hospital mortality rate was 121 (46%). In total, 7 patients were identified as having eye- or vision-related diagnosis codes, none of whom had fungal endophthalmitis based on record review. There were 26,830 free-text narrative notes associated with these 265 patients. A regular expression pipeline based on relevant terms yielded possible matches in 683 notes from 108 patients. Subsequent manual record review again demonstrated that no patients had fungal ocular involvement. Therefore, the prevalence of fungal ocular involvement in this cohort was 0%. CONCLUSIONS: MIMIC-III contained no cases of ocular involvement among fungemic patients, consistent with prior studies reporting low rates of ocular involvement in fungemia. This study demonstrates an application of natural language processing to expedite the review of narrative notes. This approach is highly relevant for ophthalmology, where diagnoses are often based on physical examination findings that are documented within clinical notes.

Latent Tuberculosis Screening Using Electronic Health Record Data.

Screening for latent tuberculosis infection is recommended for foreign-born persons in the United States. We used proxy data from electronic health records to determine that 17.5% of foreign-born outpatients attending the UC San Diego Health clinic (San Diego, CA, USA) underwent screening. Ending the global tuberculosis epidemic requires improved screening.