AI-Generated Draft Replies Integrated Into Health Records and Physicians' Electronic Communication.

Importance: Timely tests are warranted to assess the association between generative artificial intelligence (GenAI) use and physicians' work efforts. Objective: To investigate the association between GenAI-drafted replies for patient messages and physician time spent on answering messages and the length of replies. Design, Setting, and Participants: Randomized waiting list quality improvement (QI) study from June to August 2023 in an academic health system. Primary care physicians were randomized to an immediate activation group and a delayed activation group. Data were analyzed from August to November 2023. Exposure: Access to GenAI-drafted replies for patient messages. Main Outcomes and Measures: Time spent (1) reading messages, (2) replying to messages, (3) length of replies, and (4) physician likelihood to recommend GenAI drafts. The a priori hypothesis was that GenAI drafts would be associated with less physician time spent reading and replying to messages. A mixed-effects model was used. Results: Fifty-two physicians participated in this QI study, with 25 randomized to the immediate activation group and 27 randomized to the delayed activation group. A contemporary control group included 70 physicians. There were 18 female participants (72.0%) in the immediate group and 17 female participants (63.0%) in the delayed group; the median age range was 35-44 years in the immediate group and 45-54 years in the delayed group. The median (IQR) time spent reading messages in the immediate group was 26 (11-69) seconds at baseline, 31 (15-70) seconds 3 weeks after entry to the intervention, and 31 (14-70) seconds 6 weeks after entry. The delayed group's median (IQR) read time was 25 (10-67) seconds at baseline, 29 (11-77) seconds during the 3-week waiting period, and 32 (15-72) seconds 3 weeks after entry to the intervention. The contemporary control group's median (IQR) read times were 21 (9-54), 22 (9-63), and 23 (9-60) seconds in corresponding periods. The estimated association of GenAI was a 21.8% increase in read time (95% CI, 5.2% to 41.0%; P = .008), a -5.9% change in reply time (95% CI, -16.6% to 6.2%; P = .33), and a 17.9% increase in reply length (95% CI, 10.1% to 26.2%; P < .001). Participants recognized GenAI's value and suggested areas for improvement. Conclusions and Relevance: In this QI study, GenAI-drafted replies were associated with significantly increased read time, no change in reply time, significantly increased reply length, and some perceived benefits. Rigorous empirical tests are necessary to further examine GenAI's performance. Future studies should examine patient experience and compare multiple GenAIs, including those with medical training.

Generative artificial intelligence responses to patient messages in the electronic health record: early lessons learned.

Background: Electronic health record (EHR)-based patient messages can contribute to burnout. Messages with a negative tone are particularly challenging to address. In this perspective, we describe our initial evaluation of large language model (LLM)-generated responses to negative EHR patient messages and contend that using LLMs to generate initial drafts may be feasible, although refinement will be needed. Methods: A retrospective sample (n = 50) of negative patient messages was extracted from a health system EHR, de-identified, and inputted into an LLM (ChatGPT). Qualitative analyses were conducted to compare LLM responses to actual care team responses. Results: Some LLM-generated draft responses varied from human responses in relational connection, informational content, and recommendations for next steps. Occasionally, the LLM draft responses could have potentially escalated emotionally charged conversations. Conclusion: Further work is needed to optimize the use of LLMs for responding to negative patient messages in the EHR.

Patients' Perspectives on Plans Generated During Primary Care Visits and Self-Reported Adherence at 3 Months: Data From a Randomized Trial.

BACKGROUND: Effective primary care necessitates follow-up actions by the patient beyond the visit. Prior research suggests room for improvement in patient adherence. OBJECTIVE: This study sought to understand patients' views on their primary care visits, the plans generated therein, and their self-reported adherence after 3 months. METHODS: As part of a large multisite cluster randomized pragmatic trial in 3 health care organizations, patients completed 2 surveys-the first within 7 days after the index primary care visit and another 3 months later. For this analysis of secondary outcomes, we combined the results across all study participants to understand patient adherence to care plans. We recorded patient characteristics and survey responses. Cross-tabulation and chi-square statistics were used to examine bivariate associations, adjusting for multiple comparisons when appropriate. We used multivariable logistic regression to assess how patients' intention to follow, agreement, and understanding of their plans impacted their plan adherence, allowing for differences in individual characteristics. Qualitative content analysis was conducted to characterize the patient's self-reported plans and reasons for adhering (or not) to the plan 3 months later. RESULTS: Of 2555 patients, most selected the top box option (9=definitely agree) that they felt they had a clear plan (n=2011, 78%), agreed with the plan (n=2049, 80%), and intended to follow the plan (n=2108, 83%) discussed with their provider at the primary care visit. The most common elements of the plans reported included reference to exercise (n=359, 14.1%), testing (laboratory, imaging, etc; n=328, 12.8%), diet (n=296, 11.6%), and initiation or adjustment of medications; (n=284, 11.1%). Patients who strongly agreed that they had a clear plan, agreed with the plan, and intended to follow the plan were all more likely to report plan completion 3 months later (P<.001) than those providing less positive ratings. Patients who reported plans related to following up with the primary care provider (P=.008) to initiate or adjust medications (P≤.001) and to have a specialist visit were more likely to report that they had completely followed the plan (P=.003). Adjusting for demographic variables, patients who indicated intent to follow their plan were more likely to follow-through 3 months later (P<.001). Patients' reasons for completely following the plan were mainly that the plan was clear (n=1114, 69.5%), consistent with what mattered (n=1060, 66.1%), and they were determined to carry through with the plan (n=887, 53.3%). The most common reasons for not following the plan were lack of time (n=217, 22.8%), having decided to try a different approach (n=105, 11%), and the COVID-19 pandemic impacted the plan (n=105, 11%). CONCLUSIONS: Patients' initial assessment of their plan as clear, their agreement with the plan, and their initial willingness to follow the plan were all strongly related to their self-reported completion of the plan 3 months later. Patients whose plans involved lifestyle changes were less likely to report that they had "completely" followed their plan. TRIAL REGISTRATION: ClinicalTrials.gov NCT03385512; https://clinicaltrials.gov/study/NCT03385512. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.2196/30431.

Artificial Intelligence-Generated Draft Replies to Patient Inbox Messages.

Importance: The emergence and promise of generative artificial intelligence (AI) represent a turning point for health care. Rigorous evaluation of generative AI deployment in clinical practice is needed to inform strategic decision-making. Objective: To evaluate the implementation of a large language model used to draft responses to patient messages in the electronic inbox. Design, Setting, and Participants: A 5-week, prospective, single-group quality improvement study was conducted from July 10 through August 13, 2023, at a single academic medical center (Stanford Health Care). All attending physicians, advanced practice practitioners, clinic nurses, and clinical pharmacists from the Divisions of Primary Care and Gastroenterology and Hepatology were enrolled in the pilot. Intervention: Draft replies to patient portal messages generated by a Health Insurance Portability and Accountability Act-compliant electronic health record-integrated large language model. Main Outcomes and Measures: The primary outcome was AI-generated draft reply utilization as a percentage of total patient message replies. Secondary outcomes included changes in time measures and clinician experience as assessed by survey. Results: A total of 197 clinicians were enrolled in the pilot; 35 clinicians who were prepilot beta users, out of office, or not tied to a specific ambulatory clinic were excluded, leaving 162 clinicians included in the analysis. The survey analysis cohort consisted of 73 participants (45.1%) who completed both the presurvey and postsurvey. In gastroenterology and hepatology, there were 58 physicians and APPs and 10 nurses. In primary care, there were 83 physicians and APPs, 4 nurses, and 8 clinical pharmacists. The mean AI-generated draft response utilization rate across clinicians was 20%. There was no change in reply action time, write time, or read time between the prepilot and pilot periods. There were statistically significant reductions in the 4-item physician task load score derivative (mean [SD], 61.31 [17.23] presurvey vs 47.26 [17.11] postsurvey; paired difference, -13.87; 95% CI, -17.38 to -9.50; P < .001) and work exhaustion scores (mean [SD], 1.95 [0.79] presurvey vs 1.62 [0.68] postsurvey; paired difference, -0.33; 95% CI, -0.50 to -0.17; P < .001). Conclusions and Relevance: In this quality improvement study of an early implementation of generative AI, there was notable adoption, usability, and improvement in assessments of burden and burnout. There was no improvement in time. Further code-to-bedside testing is needed to guide future development and organizational strategy.

Navigating the doctor-patient-AI relationship - a mixed-methods study of physician attitudes toward artificial intelligence in primary care.

BACKGROUND: Artificial intelligence (AI) is a rapidly advancing field that is beginning to enter the practice of medicine. Primary care is a cornerstone of medicine and deals with challenges such as physician shortage and burnout which impact patient care. AI and its application via digital health is increasingly presented as a possible solution. However, there is a scarcity of research focusing on primary care physician (PCP) attitudes toward AI. This study examines PCP views on AI in primary care. We explore its potential impact on topics pertinent to primary care such as the doctor-patient relationship and clinical workflow. By doing so, we aim to inform primary care stakeholders to encourage successful, equitable uptake of future AI tools. Our study is the first to our knowledge to explore PCP attitudes using specific primary care AI use cases rather than discussing AI in medicine in general terms. METHODS: From June to August 2023, we conducted a survey among 47 primary care physicians affiliated with a large academic health system in Southern California. The survey quantified attitudes toward AI in general as well as concerning two specific AI use cases. Additionally, we conducted interviews with 15 survey respondents. RESULTS: Our findings suggest that PCPs have largely positive views of AI. However, attitudes often hinged on the context of adoption. While some concerns reported by PCPs regarding AI in primary care focused on technology (accuracy, safety, bias), many focused on people-and-process factors (workflow, equity, reimbursement, doctor-patient relationship). CONCLUSION: Our study offers nuanced insights into PCP attitudes towards AI in primary care and highlights the need for primary care stakeholder alignment on key issues raised by PCPs. AI initiatives that fail to address both the technological and people-and-process concerns raised by PCPs may struggle to make an impact.

Measures of empathy and compassion: A scoping review.

Evidence to date indicates that compassion and empathy are health-enhancing qualities. Research points to interventions and practices involving compassion and empathy being beneficial, as well as being salient outcomes of contemplative practices such as mindfulness. Advancing the science of compassion and empathy requires that we select measures best suited to evaluating effectiveness of training and answering research questions. The objective of this scoping review was to 1) determine what instruments are currently available for measuring empathy and compassion, 2) assess how and to what extent they have been validated, and 3) provide an online tool to assist researchers and program evaluators in selecting appropriate measures for their settings and populations. A scoping review and broad evidence map were employed to systematically search and present an overview of the large and diverse body of literature pertaining to measuring compassion and empathy. A search string yielded 19,446 articles, and screening resulted in 559 measure development or validation articles reporting on 503 measures focusing on or containing subscales designed to measure empathy and/or compassion. For each measure, we identified the type of measure, construct being measured, in what context or population it was validated, response set, sample items, and how many different types of psychometrics had been assessed for that measure. We provide tables summarizing these data, as well as an open-source online interactive data visualization allowing viewers to search for measures of empathy and compassion, review their basic qualities, and access original citations containing more detail. Finally, we provide a rubric to help readers determine which measure(s) might best fit their context.

Physician Turnover in the United States.

BACKGROUND: Medical groups, health systems, and professional associations are concerned about potential increases in physician turnover, which may affect patient access and quality of care. OBJECTIVE: To examine whether turnover has changed over time and whether it is higher for certain types of physicians or practice settings. DESIGN: The authors developed a novel method using 100% of traditional Medicare billing to create national estimates of turnover. Standardized turnover rates were compared by physician, practice, and patient characteristics. SETTING: Traditional Medicare, 2010 to 2020. PARTICIPANTS: Physicians billing traditional Medicare. MEASUREMENTS: Indicators of physician turnover-physicians who stopped practicing and those who moved from one practice to another-and their sum. RESULTS: The annual rate of turnover increased from 5.3% to 7.2% between 2010 and 2014, was stable through 2017, and increased modestly in 2018 to 7.6%. Most of the increase from 2010 to 2014 came from physicians who stopped practicing increasing from 1.6% to 3.1%; physicians moving increased modestly from 3.7% to 4.2%. Modest but statistically significant (P < 0.001) differences existed across rurality, physician sex, specialty, and patient characteristics. In the second and third quarters of 2020, quarterly turnover was slightly lower than in the corresponding quarters of 2019. LIMITATION: Measurement was based on traditional Medicare claims. CONCLUSION: Over the past decade, physician turnover rates have had periods of increase and stability. These early data, covering the first 3 quarters of 2020, give no indication yet of the COVID-19 pandemic increasing turnover, although continued tracking of turnover is warranted. This novel method will enable future monitoring and further investigations into turnover. PRIMARY FUNDING SOURCE: The Physicians Foundation Center for the Study of Physician Practice and Leadership.

Association of physician burnout with perceived EHR work stress and potentially actionable factors.

OBJECTIVE: Physicians of all specialties experienced unprecedented stressors during the COVID-19 pandemic, exacerbating preexisting burnout. We examine burnout's association with perceived and actionable electronic health record (EHR) workload factors and personal, professional, and organizational characteristics with the goal of identifying levers that can be targeted to address burnout. MATERIALS AND METHODS: Survey of physicians of all specialties in an academic health center, using a standard measure of burnout, self-reported EHR work stress, and EHR-based work assessed by the number of messages regarding prescription reauthorization and use of a staff pool to triage messages. Descriptive and multivariable regression analyses examined the relationship among burnout, perceived EHR work stress, and actionable EHR work factors. RESULTS: Of 1038 eligible physicians, 627 responded (60% response rate), 49.8% reported burnout symptoms. Logistic regression analysis suggests that higher odds of burnout are associated with physicians feeling higher level of EHR stress (odds ratio [OR], 1.15; 95% confidence interval [CI], 1.07-1.25), having more prescription reauthorization messages (OR, 1.23; 95% CI, 1.04-1.47), not feeling valued (OR, 3.38; 95% CI, 1.69-7.22) or aligned in values with clinic leaders (OR, 2.81; 95% CI, 1.87-4.27), in medical practice for ≤15 years (OR, 2.57; 95% CI, 1.63-4.12), and sleeping for <6 h/night (OR, 1.73; 95% CI, 1.12-2.67). DISCUSSION: Perceived EHR stress and prescription reauthorization messages are significantly associated with burnout, as are non-EHR factors such as not feeling valued or aligned in values with clinic leaders. Younger physicians need more support. CONCLUSION: A multipronged approach targeting actionable levers and supporting young physicians is needed to implement sustainable improvements in physician well-being.

"Mm-hm," "Uh-uh": are non-lexical conversational sounds deal breakers for the ambient clinical documentation technology?

OBJECTIVES: Ambient clinical documentation technology uses automatic speech recognition (ASR) and natural language processing (NLP) to turn patient-clinician conversations into clinical documentation. It is a promising approach to reducing clinician burden and improving documentation quality. However, the performance of current-generation ASR remains inadequately validated. In this study, we investigated the impact of non-lexical conversational sounds (NLCS) on ASR performance. NLCS, such as Mm-hm and Uh-uh, are commonly used to convey important information in clinical conversations, for example, Mm-hm as a "yes" response from the patient to the clinician question "are you allergic to antibiotics?" MATERIALS AND METHODS: In this study, we evaluated 2 contemporary ASR engines, Google Speech-to-Text Clinical Conversation ("Google ASR"), and Amazon Transcribe Medical ("Amazon ASR"), both of which have their language models specifically tailored to clinical conversations. The empirical data used were from 36 primary care encounters. We conducted a series of quantitative and qualitative analyses to examine the word error rate (WER) and the potential impact of misrecognized NLCS on the quality of clinical documentation. RESULTS: Out of a total of 135 647 spoken words contained in the evaluation data, 3284 (2.4%) were NLCS. Among these NLCS, 76 (0.06% of total words, 2.3% of all NLCS) were used to convey clinically relevant information. The overall WER, of all spoken words, was 11.8% for Google ASR and 12.8% for Amazon ASR. However, both ASR engines demonstrated poor performance in recognizing NLCS: the WERs across frequently used NLCS were 40.8% (Google) and 57.2% (Amazon), respectively; and among the NLCS that conveyed clinically relevant information, 94.7% and 98.7%, respectively. DISCUSSION AND CONCLUSION: Current ASR solutions are not capable of properly recognizing NLCS, particularly those that convey clinically relevant information. Although the volume of NLCS in our evaluation data was very small (2.4% of the total corpus; and for NLCS that conveyed clinically relevant information: 0.06%), incorrect recognition of them could result in inaccuracies in clinical documentation and introduce new patient safety risks.

Unmet needs for food, medicine, and mental health services among vulnerable older adults during the COVID-19 pandemic.

OBJECTIVE: To examine sociodemographic factors associated with having unmet needs in medications, mental health, and food security among older adults during the COVID-19 pandemic. DATA SOURCES AND STUDY SETTING: Primary data and secondary data from the electronic health records (EHR) in an age-friendly academic health system in 2020 were used. STUDY DESIGN: Observational study examining factors associated with having unmet needs in medications, food, and mental health. DATA COLLECTING/EXTRACTION METHODS: Data from a computer-assisted telephone interview and EHR on community-dwelling older patients were analyzed. PRINCIPLE FINDINGS: Among 3400 eligible patients, 1921 (53.3%) (average age 76, SD 11) responded, with 857 (45%) of respondents having at least one unmet need. Unmet needs for medications were present in 595 (31.0%), for food in 196 (10.2%), and for mental health services in 292 (15.2%). Racial minorities had significantly higher probabilities of having unmet needs for medicine and food, and of being referred for services related to medications, food, and mental health. Patients living in more resource-limited neighborhoods had a higher probability of being referred for mental health services. CONCLUSIONS: Age-friendly health systems (AFHS) and their recognition should include assessing and addressing social risk factors among older adults. Proactive efforts to address unmet needs should be integral to AFHS.

Association of Electronic Health Record Inbasket Message Characteristics With Physician Burnout.

Importance: Physician burnout is an ongoing epidemic; electronic health record (EHR) use has been associated with burnout, and the burden of EHR inbasket messages has grown in the context of the COVID-19 pandemic. Understanding how EHR inbasket messages are associated with physician burnout may uncover new insights for intervention strategies. Objective: To evaluate associations between EHR inbasket message characteristics and physician burnout. Design, Setting, and Participants: Cross-sectional study in a single academic medical center involving physicians from multiple specialties. Data collection took place April to September 2020, and data were analyzed September to December 2020. Exposures: Physicians responded to a survey including the validated Mini-Z 5-point burnout scale. Main Outcomes and Measures: Physician burnout according to the self-reported burnout scale. A sentiment analysis model was used to calculate sentiment scores for EHR inbasket messages extracted for participating physicians. Multivariable modeling was used to model risk of physician burnout using factors such as message characteristics, physician demographics, and clinical practice characteristics. Results: Of 609 physicians who responded to the survey, 297 (48.8%) were women, 343 (56.3%) were White, 391 (64.2%) practiced in outpatient settings, and 428 (70.28%) had been in medical practice for 15 years or less. Half (307 [50.4%]) reported burnout (score of 3 or higher). A total of 1 453 245 inbasket messages were extracted, of which 630 828 (43.4%) were patient messages. Among negative messages, common words included medical conditions, expletives and/or profanity, and words related to violence. There were no significant associations between message characteristics (including sentiment scores) and burnout. Odds of burnout were significantly higher among Hispanic/Latino physicians (odds ratio [OR], 3.44; 95% CI, 1.18-10.61; P = .03) and women (OR, 1.60; 95% CI, 1.13-2.27; P = .01), and significantly lower among physicians in clinical practice for more than 15 years (OR, 0.46; 95% CI, 0.30-0.68; P < .001). Conclusions and Relevance: In this cross-sectional study, message characteristics were not associated with physician burnout, but the presence of expletives and violent words represents an opportunity for improving patient engagement, EHR portal design, or filters. Natural language processing represents a novel approach to understanding potential associations between EHR inbasket messages and physician burnout and may also help inform quality improvement initiatives aimed at improving patient experience.

Population-Based Evaluation of Postacute Coronavirus Disease-2019 (COVID-19) Chronic Sequelae in Patients Who Tested Positive for Severe Acute Respiratory Syndrome Coronavirus-2 (SARS-CoV-2).

The true incidence and comprehensive characteristics of Long Coronavirus Disease-19 (COVID-19) are currently unknown. This is the first population-based outreach study of Long COVID within an entire health system, conducted to determine operational needs to care for patients with Long COVID.

Relationship between burnout and Major Depressive Disorder in health professionals: A HEAR report.

BACKGROUND: Burnout is a "normal" albeit concerning response to workplace stress, whereas Major Depressive Disorder (MDD) is a serious illness associated with impairment and suicide risk. Because of symptomatic overlap between the two conditions and MDD-associated stigma, individuals reporting work-related stress and depression often are "diagnosed" with burnout at the expense of recognizing and treating MDD. Our study aimed to leverage organizational implementation of the American Foundation of Suicide Prevention's Interactive Screening Program to elucidate relationships among burnout, depression, and other suicide risk factors. METHODS: 2281 of about 30,000 (~7.6 %) medical trainees, staff, and faculty responded to an anonymous online stress and depression questionnaire. Respondents were grouped into four cohorts: screened positive for burnout alone (n = 439, 19 %), depression alone (n = 268, 12 %), both conditions (n = 759, 33 %), or neither condition (n = 817, 36 %), and compared on multiple measures of distress and other suicide risk factors. RESULTS: Burnout alone and depression alone each predicted greater distress and suicide risk compared with neither condition. Depression was a stronger predictor than burnout and demonstrated a consistent association with other suicide risk factors regardless of whether burnout was present. In contrast, burnout was not consistently associated with other suicide risk factors when depression was present. LIMITATIONS: The sample was limited to one state-supported academic medical center; to individuals who elected to take the online survey; and relied on a single item, non-validated measure of burnout. CONCLUSION: When emotional distress is reported by healthcare workers, attention should not stop at "burnout," as burnout frequently comingles with clinical depression, a serious and treatable mental health condition.

COVID-19 vaccine hesitancy: Associations with gender, race, and source of health information.

INTRODUCTION: Vaccinations for COVID-19 are being distributed, yet vaccine hesitance is placing many people at risk for infection, negative outcomes, and compromising public health. Given primary care clinics are where people most often interact with health care providers, understanding factors associated with this hesitance may help providers in integrated primary care settings best address this hesitance. METHOD: Between September and November of 2020, a survey was sent to all primary care patients within a large southern California health system, with over 10,000 responding (22% response rate). Survey items included sociodemographic variables, level of vaccine hesitance, "proximity to COVID" (e.g., direct exposure to COVID-19 and consequences), as well as a patient's primary source of health information (e.g., traditional news, social media, etc.). Responses assessed the strength of hesitance. RESULTS: Results showed that while 78% of participants "strongly" believed vaccines generally are a good way to protect from illness, only 51% reported strong willingness to get the COVID-19 vaccine. Consistent with previous surveys, younger patients were more hesitant to get vaccinated, as were people of color. Unique to this survey was the finding that those relying on social media, faith-based organizations, or family/friends for health information had the greatest vaccine hesitance. DISCUSSION: While our patient sample was less hesitant than other U.S. adult samples previously reported in the literature, our data suggest that targeting those patients who report reliance on nontraditional health information sources should be approached by primary care teams, including behavioral health providers, to address vaccine hesitancy. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Comparing the Use of DynaMed and UpToDate by Physician Trainees in Clinical Decision-Making: A Randomized Crossover Trial.

BACKGROUND: Costs vary substantially among electronic medical knowledge resources used for clinical decision support, warranting periodic assessment of institution-wide adoption. OBJECTIVES: To compare two medical knowledge resources, UpToDate and DynaMed Plus, regarding accuracy and time required to answer standardized clinical questions and user experience. METHODS: A crossover trial design was used, wherein physicians were randomized to first use one of the two medical knowledge resources to answer six standardized questions. Following use of each resource, they were surveyed regarding their user experience. The percentage of accurate answers and time required to answer each question were recorded. The surveys assessed ease of use, enjoyment using the resource, quality of information, and ability to assess level of evidence. Tests of carry-over effects were performed. Themes were identified within open-ended survey comments regarding overall user experience. RESULTS: Among 26 participating physicians, accuracy of answers differed by 4 percentage points or less. For all but one question, there were no significant differences in the time required for completion. Most participants felt both resources were easy to use, contained high quality of information, and enabled assessment of the level of evidence. A greater proportion of participants endorsed enjoyment of use with UpToDate (23/26, 88%) compared with DynaMed Plus (16/26, 62%). Themes from open-ended comments included interface/information presentation, coverage of clinical topics, search functions, and utility for clinical decision-making. The majority (59%) of open-ended comments expressed an overall preference for UpToDate, compared with 19% preferring DynaMed Plus. CONCLUSION: DynaMed Plus is noninferior to UpToDate with respect to ability to achieve accurate answers, time required for answering clinical questions, ease of use, quality of information, and ability to assess level of evidence. However, user experience was more positive with UpToDate. Future studies of electronic medical knowledge resources should continue to emphasize evaluation of usability and user experience.

A learning health system approach to COVID-19 exposure notification system rollout.

Introduction: Digital exposure notification (EN) approaches may offer considerable advantages over traditional contact tracing in speed, scale, efficacy, and confidentiality in pandemic control. We applied the science of learning health systems to test the effect of framing and digital means, email vs Short Message Service (SMS), on EN adoption among patients of an academic health center. Methods: We tested three communication approaches of the Apple and Google EN system in a rapid learning cycle involving 15 000 patients pseudorandomly assigned to three groups. The patients in the first group received a 284-word email that presented EN as a tool that can help slow the spread. The patients in the second group received a 32-word SMS that described EN as a new tool to help slow the spread (SlowTheSpreadSMS). Patients in the third group received a 47-word SMS that depicted the system as a new digital tool that can empower them to protect their family and friends (EmpowerSMS). A brief four-question anonymous survey of adoption was included in a reminder message sent 2 days after the initial outreach. Results: One hundred and sixty people responded to the survey within 1 week: 2.33% from EmpowerSMS, 0.97% from SlowTheSpreadSMS, and 0.53% from emails; 29 (41.43%), 24 (41.38%), and 11 (34.38%) reported having adopted EN from each group, respectively. Patient reported barriers to adoption included iOS version incompatibility, privacy concerns, and low trust of government agencies or companies like Apple and Google. Patients recommended that healthcare systems play an active role in disseminating information about this tool. Patients also recommended advertising on social media and providing reassurance about privacy. Conclusions: The EmpowerSMS resulted in relatively more survey responses. Both SMS groups had slightly higher, but not statistically significant EN adoption rates compared to email. Findings from the pilot not only informed operational decision-making in our health system but also contributed to EN rollout planning in our State.

Automatic speech recognition performance for digital scribes: a performance comparison between general-purpose and specialized models tuned for patient-clinician conversations.

One promising solution to address physician data entry needs is through the development of so-called "digital scribes," or tools which aim to automate clinical documentation via automatic speech recognition (ASR) of patient-clinician conversations. Evaluation of specialized ASR models in this domain, useful for understanding feasibility and development opportunities, has been difficult because most models have been under development. Following the commercial release of such models, we report an independent evaluation of four models, two general-purpose, and two for medical conversation with a corpus of 36 primary care conversations. We identify word error rates (WER) of 8.8%-10.5% and word-level diarization error rates (WDER) ranging from 1.8%-13.9%, which are generally lower than previous reports. The findings indicate that, while there is room for improvement, the performance of these specialized models, at least under ideal recording conditions, may be amenable to the development of downstream applications which rely on ASR of patient-clinician conversations.

Implementation of Patient Engagement Tools in Electronic Health Records to Enhance Patient-Centered Communication: Protocol for Feasibility Evaluation and Preliminary Results.

BACKGROUND: Patient-physician communication during clinical encounters is essential to ensure quality of care. Many studies have attempted to improve patient-physician communication. Incorporating patient priorities into agenda setting and medical decision-making are fundamental to patient-centered communication. Efficient and scalable approaches are needed to empower patients to speak up and prepare physicians to respond. Leveraging electronic health records (EHRs) in engaging patients and health care teams has the potential to enhance the integration of patient priorities in clinical encounters. A systematic approach to eliciting and documenting patient priorities before encounters could facilitate effective communication in such encounters. OBJECTIVE: In this paper, we report the design and implementation of a set of EHR tools built into clinical workflows for facilitating patient-physician joint agenda setting and the documentation of patient concerns in the EHRs for ambulatory encounters. METHODS: We engaged health information technology leaders and users in three health care systems for developing and implementing a set of EHR tools. The goal of these tools is to standardize the elicitation of patient priorities by using a previsit "patient important issue" questionnaire distributed through the patient portal to the EHR. We built additional EHR documentation tools to facilitate patient-staff communication when the staff records the vital signs and the reason for the visit in the EHR while in the examination room, with a simple transmission method for physicians to incorporate patient concerns in EHR notes. RESULTS: The study is ongoing. The anticipated completion date for survey data collection is November 2021. A total of 34,037 primary care patients from three health systems (n=26,441; n=5136; and n=2460 separately recruited from each system) used the previsit patient important issue questionnaire in 2020. The adoption of the digital previsit questionnaire during the COVID-19 pandemic was much higher in one health care system because it expanded the use of the questionnaire from physicians participating in trials to all primary care providers midway through the year. It also required the use of this previsit questionnaire for eCheck-ins, which are required for telehealth encounters. Physicians and staff suggested anecdotally that this questionnaire helped patient-clinician communication, particularly during the COVID-19 pandemic. CONCLUSIONS: EHR tools have the potential to facilitate the integration of patient priorities into agenda setting and documentation in real-world primary care practices. Early results suggest the feasibility and acceptability of such digital tools in three health systems. EHR tools can support patient engagement and clinicians' work during in-person and telehealth visits. They could potentially exert a sustained influence on patient and clinician communication behaviors in contrast to prior ad hoc educational efforts targeting patients or clinicians. TRIAL REGISTRATION: ClinicalTrials.gov NCT03385512; https://clinicaltrials.gov/ct2/show/NCT03385512. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/30431.

Measuring Public Reaction to Violence Against Doctors in China: Interrupted Time Series Analysis of Media Reports.

BACKGROUND: Violence against doctors in China is a serious problem that has attracted attention from both domestic and international media. OBJECTIVE: This study investigates readers' responses to media reports on violence against doctors to identify attitudes toward perpetrators and physicians and examine if such trends are influenced by national policies. METHODS: We searched 17 Chinese violence against doctors reports in international media sources from 2011 to 2020. We then tracked back the original reports and web crawled the 19,220 comments in China. To ascertain the possible turning point of public opinion, we searched violence against doctors-related policies from Tsinghua University ipolicy database from 2011 to 2020, and found 19 policies enacted by the Chinese central government aimed at alleviating the intense patient-physician relationship. We then conducted a series of interrupted time series analyses to examine the influence of these policies on public sentiment toward violence against doctors over time. RESULTS: The interrupted time series analysis (ITSA) showed that the change in public sentiment toward violence against doctors reports was temporally associated with government interventions. The declarations of 10 of the public policies were followed by increases in the proportion of online public opinion in support of doctors (average slope changes of 0.010, P<.05). A decline in the proportion of online public opinion that blamed doctors (average level change of -0.784, P<.05) followed the declaration of 3 policies. CONCLUSIONS: The government's administrative interventions effectively shaped public opinion but only temporarily. Continued public policy interventions are needed to sustain the reduction of hostility toward medical doctors.

Automated rating of patient and physician emotion in primary care visits.

OBJECTIVE: Train machine learning models that automatically predict emotional valence of patient and physician in primary care visits. METHODS: Using transcripts from 353 primary care office visits with 350 patients and 84 physicians (Cook, 2002 [1], Tai-Seale et al., 2015 [2]), we developed two machine learning models (a recurrent neural network with a hierarchical structure and a logistic regression classifier) to recognize the emotional valence (positive, negative, neutral) (Posner et al., 2005 [3]) of each utterance. We examined the agreement of human-generated ratings of emotional valence with machine learning model ratings of emotion. RESULTS: The agreement of emotion ratings from the recurrent neural network model with human ratings was comparable to that of human-human inter-rater agreement. The weighted-average of the correlation coefficients for the recurrent neural network model with human raters was 0.60, and the human rater agreement was also 0.60. CONCLUSIONS: The recurrent neural network model predicted the emotional valence of patients and physicians in primary care visits with similar reliability as human raters. PRACTICE IMPLICATIONS: As the first machine learning-based evaluation of emotion recognition in primary care visit conversations, our work provides valuable baselines for future applications that might help monitor patient emotional signals, supporting physicians in empathic communication, or examining the role of emotion in patient-centered care.