Implementing a Clinical Decision Support Tool to Improve Physical Activity.

BACKGROUND: Currently, only about half of U.S. adults achieve current physical activity guidelines. Routine physical activity is not regularly assessed, nor are patients routinely counseled by their healthcare provider on achieving recommended levels. The three-question physical activity vital sign (PAVS) was developed to assess physical activity duration and intensity and identify adults not meeting physical activity guidelines. Clinical decision support provided via a best practice advisory in an electronic health record (EHR) system can be triggered as a prompt, reminding healthcare providers to implement the best practice intervention when appropriate. Remote patient monitoring of physical activity can provide objective data in the EHR. OBJECTIVES: This study aimed to evaluate the feasibility and clinical utility of embedding the PAVS and a triggered best practice advisor into the EHR in an ambulatory preventive cardiology practice setting to alert providers to patients reporting low physical activity and prompt healthcare providers to counsel these patients as needed. METHODS: Three components based in the EHR were integrated for the purpose of this study: Patients completed the PAVS through their electronic patient portal prior to an office visit, a best practice advisory was created to prompt providers to counsel patients who reported low levels of physical activity, and remote patient monitoring via Fitbit synced to the EHR provided objective physical activity data. The intervention was pilot-tested in the Epic EHR for 1 year (July 1, 2021 to June 30, 2022). Qualitative feedback on the intervention from both providers and patients was obtained at the completion of the study. RESULTS: Monthly assessments of the use of the PAVS and best practice advisory and remote patient monitoring were completed. Patients' completion of the PAVS varied from 35% to 48% per month. The best practice advisory was signed by providers between 2% and 65% and was acknowledged by 2%-22% per month. The majority (58%) of patients were able to sync a Fitbit device to their EHR for remote monitoring. DISCUSSION: Although uptake of each component needs improvement, this pilot demonstrated the feasibility of incorporating a physical activity promotion intervention into the EHR. Qualitative feedback provided guidance for future implementation.

Remote patient monitoring for management of diabetes mellitus in pregnancy is associated with improved maternal and neonatal outcomes.

BACKGROUND: Diabetes mellitus is a common medical complication of pregnancy, and its treatment is complex. Recent years have seen an increase in the application of mobile health tools and advanced technologies, such as remote patient monitoring, with the aim of improving care for diabetes mellitus in pregnancy. Previous studies of these technologies for the treatment of diabetes in pregnancy have been small and have not clearly shown clinical benefit with implementation. OBJECTIVE: Remote patient monitoring allows clinicians to monitor patients' health data (such as glucose values) in near real-time, between office visits, to make timely adjustments to care. Our objective was to determine if using remote patient monitoring for the management of diabetes in pregnancy leads to an improvement in maternal and neonatal outcomes. STUDY DESIGN: This was a retrospective cohort study of pregnant patients with diabetes mellitus managed by the maternal-fetal medicine practice at one academic institution between October 2019 and April 2021. This practice transitioned from paper-based blood glucose logs to remote patient monitoring in February 2020. Remote patient monitoring options included (1) device integration with Bluetooth glucometers that automatically uploaded measured glucose values to the patient's Epic MyChart application or (2) manual entry in which patients manually logged their glucose readings into their MyChart application. Values in the MyChart application directly transferred to the patient's electronic health record for review and management by clinicians. In total, 533 patients were studied. We compared 173 patients managed with paper logs to 360 patients managed with remote patient monitoring (176 device integration and 184 manual entry). Our primary outcomes were composite maternal morbidity (which included third- and fourth-degree lacerations, chorioamnionitis, postpartum hemorrhage requiring transfusion, postpartum hysterectomy, wound infection or separation, venous thromboembolism, and maternal admission to the intensive care unit) and composite neonatal morbidity (which included umbilical cord pH <7.00, 5 minute Apgar score <7, respiratory morbidity, hyperbilirubinemia, meconium aspiration, intraventricular hemorrhage, necrotizing enterocolitis, sepsis, pneumonia, seizures, hypoxic ischemic encephalopathy, shoulder dystocia, trauma, brain or body cooling, and neonatal intensive care unit admission). Secondary outcomes were measures of glycemic control and the individual components of the primary composite outcomes. We also performed a secondary analysis in which the patients who used the two different remote patient monitoring options (device integration vs manual entry) were compared. Chi-square, Fisher's exact, 2-sample t, and Mann-Whitney tests were used to compare the groups. A result was considered statistically significant at P<.05. RESULTS: Maternal baseline characteristics were not significantly different between the remote patient monitoring and paper groups aside from a slightly higher baseline rate of chronic hypertension in the remote patient monitoring group (6.1% vs 1.2%; P=.011). The primary outcomes of composite maternal and composite neonatal morbidity were not significantly different between the groups. However, remote patient monitoring patients submitted more glucose values (177 vs 146; P=.008), were more likely to achieve glycemic control in target range (79.2% vs 52.0%; P<.0001), and achieved the target range sooner (median, 3.3 vs 4.1 weeks; P=.025) than patients managed with paper logs. This was achieved without increasing in-person visits. Remote patient monitoring patients had lower rates of preeclampsia (5.8% vs 15.0%; P=.0006) and their infants had lower rates of neonatal hypoglycemia in the first 24 hours of life (29.8% vs 51.7%; P<.0001). CONCLUSION: Remote patient monitoring for the management of diabetes mellitus in pregnancy is superior to a traditional paper-based approach in achieving glycemic control and is associated with improved maternal and neonatal outcomes.

Considerations for using predictive models that include race as an input variable: The case study of lung cancer screening.

Indiscriminate use of predictive models incorporating race can reinforce biases present in source data and lead to an exacerbation of health disparities. In some countries, such as the United States, there is therefore a push to remove race from prediction models; however, there are still many prediction models that use race as an input. Biomedical informaticists who are given the responsibility of using these predictive models in healthcare environments are likely to be faced with questions like how to deal with race covariates in these models. Thus, there is a need for a pragmatic framework to help model users think through how to include race in their chosen model so as to avoid inadvertently exacerbating disparities. In this paper, we use the case study of lung cancer screening to propose a simple framework to guide how model users can approach the use (or non-use) of race inputs in the predictive models they are tasked with leveraging in electronic health records and clinical workflows.

Reducing prescribing of antibiotics for acute respiratory infections using a frontline nurse-led EHR-Integrated clinical decision support tool: protocol for a stepped wedge randomized control trial.

BACKGROUND: Overprescribing of antibiotics for acute respiratory infections (ARIs) remains a major issue in outpatient settings. Use of clinical prediction rules (CPRs) can reduce inappropriate antibiotic prescribing but they remain underutilized by physicians and advanced practice providers. A registered nurse (RN)-led model of an electronic health record-integrated CPR (iCPR) for low-acuity ARIs may be an effective alternative to address the barriers to a physician-driven model. METHODS: Following qualitative usability testing, we will conduct a stepped-wedge practice-level cluster randomized controlled trial (RCT) examining the effect of iCPR-guided RN care for low acuity patients with ARI. The primary hypothesis to be tested is: Implementation of RN-led iCPR tools will reduce antibiotic prescribing across diverse primary care settings. Specifically, this study aims to: (1) determine the impact of iCPRs on rapid strep test and chest x-ray ordering and antibiotic prescribing rates when used by RNs; (2) examine resource use patterns and cost-effectiveness of RN visits across diverse clinical settings; (3) determine the impact of iCPR-guided care on patient satisfaction; and (4) ascertain the effect of the intervention on RN and physician burnout. DISCUSSION: This study represents an innovative approach to using an iCPR model led by RNs and specifically designed to address inappropriate antibiotic prescribing. This study has the potential to provide guidance on the effectiveness of delegating care of low-acuity patients with ARIs to RNs to increase use of iCPRs and reduce antibiotic overprescribing for ARIs in outpatient settings. TRIAL REGISTRATION: ClinicalTrials.gov Identifier: NCT04255303, Registered February 5 2020, https://clinicaltrials.gov/ct2/show/NCT04255303 .

The Impact of an Electronic Best Practice Advisory on Patients' Physical Activity and Cardiovascular Risk Profile.

BACKGROUND: Regular physical activity (PA) is a component of cardiovascular health and is associated with a lower risk of cardiovascular disease (CVD). However, only about half of US adults achieved the current PA recommendations. OBJECTIVE: The study purpose was to implement PA counseling using a clinical decision support tool in a preventive cardiology clinic and to assess changes in CVD risk factors in a sample of patients enrolled over 12 weeks of PA monitoring. METHODS: This intervention, piloted for 1 year, had 3 components embedded in the electronic health record: assessment of patients' PA, an electronic prompt for providers to counsel patients reporting low PA, and patient monitoring using a Fitbit. Cardiovascular disease risk factors included PA (self-report and Fitbit), body mass index, blood pressure, lipids, and cardiorespiratory fitness assessed with the 6-minute walk test. Depression and quality of life were also assessed. Paired t tests assessed changes in CVD risk. RESULTS: The sample who enrolled in the remote patient monitoring (n = 59) were primarily female (51%), White adults (76%) with a mean age of 61.13 ± 11.6 years. Self-reported PA significantly improved over 12 weeks (P = .005), but not Fitbit steps (P = .07). There was a significant improvement in cardiorespiratory fitness (469 ± 108 vs 494 ± 132 m, P = .0034), and 23 participants (42%) improved at least 25 m, signifying a clinically meaningful improvement. Only 4 participants were lost to follow-up over 12 weeks of monitoring. CONCLUSIONS: Patients may need more frequent reminders to be active after an initial counseling session, perhaps getting automated messages based on their step counts syncing to their electronic health record.

Comparing models of delivery for cancer genetics services among patients receiving primary care who meet criteria for genetic evaluation in two healthcare systems: BRIDGE randomized controlled trial.

BACKGROUND: Advances in genetics and sequencing technologies are enabling the identification of more individuals with inherited cancer susceptibility who could benefit from tailored screening and prevention recommendations. While cancer family history information is used in primary care settings to identify unaffected patients who could benefit from a cancer genetics evaluation, this information is underutilized. System-level population health management strategies are needed to assist health care systems in identifying patients who may benefit from genetic services. In addition, because of the limited number of trained genetics specialists and increasing patient volume, the development of innovative and sustainable approaches to delivering cancer genetic services is essential. METHODS: We are conducting a randomized controlled trial, entitled Broadening the Reach, Impact, and Delivery of Genetic Services (BRIDGE), to address these needs. The trial is comparing uptake of genetic counseling, uptake of genetic testing, and patient adherence to management recommendations for automated, patient-directed versus enhanced standard of care cancer genetics services delivery models. An algorithm-based system that utilizes structured cancer family history data available in the electronic health record (EHR) is used to identify unaffected patients who receive primary care at the study sites and meet current guidelines for cancer genetic testing. We are enrolling eligible patients at two healthcare systems (University of Utah Health and New York University Langone Health) through outreach to a randomly selected sample of 2780 eligible patients in the two sites, with 1:1 randomization to the genetic services delivery arms within sites. Study outcomes are assessed through genetics clinic records, EHR, and two follow-up questionnaires at 4 weeks and 12 months after last genetic counseling contactpre-test genetic counseling. DISCUSSION: BRIDGE is being conducted in two healthcare systems with different clinical structures and patient populations. Innovative aspects of the trial include a randomized comparison of a chatbot-based genetic services delivery model to standard of care, as well as identification of at-risk individuals through a sustainable EHR-based system. The findings from the BRIDGE trial will advance the state of the science in identification of unaffected patients with inherited cancer susceptibility and delivery of genetic services to those patients. TRIAL REGISTRATION: BRIDGE is registered as NCT03985852 . The trial was registered on June 6, 2019 at clinicaltrials.gov .

Applying A/B Testing to Clinical Decision Support: Rapid Randomized Controlled Trials.

BACKGROUND: Clinical decision support (CDS) is a valuable feature of electronic health records (EHRs) designed to improve quality and safety. However, due to the complexities of system design and inconsistent results, CDS tools may inadvertently increase alert fatigue and contribute to physician burnout. A/B testing, or rapid-cycle randomized tests, is a useful method that can be applied to the EHR in order to rapidly understand and iteratively improve design choices embedded within CDS tools. OBJECTIVE: This paper describes how rapid randomized controlled trials (RCTs) embedded within EHRs can be used to quickly ascertain the superiority of potential CDS design changes to improve their usability, reduce alert fatigue, and promote quality of care. METHODS: A multistep process combining tools from user-centered design, A/B testing, and implementation science was used to understand, ideate, prototype, test, analyze, and improve each candidate CDS. CDS engagement metrics (alert views, acceptance rates) were used to evaluate which CDS version is superior. RESULTS: To demonstrate the impact of the process, 2 experiments are highlighted. First, after multiple rounds of usability testing, a revised CDS influenza alert was tested against usual care CDS in a rapid (~6 weeks) RCT. The new alert text resulted in minimal impact on reducing firings per patients per day, but this failure triggered another round of review that identified key technical improvements (ie, removal of dismissal button and firings in procedural areas) that led to a dramatic decrease in firings per patient per day (23.1 to 7.3). In the second experiment, the process was used to test 3 versions (financial, quality, regulatory) of text supporting tobacco cessation alerts as well as 3 supporting images. Based on 3 rounds of RCTs, there was no significant difference in acceptance rates based on the framing of the messages or addition of images. CONCLUSIONS: These experiments support the potential for this new process to rapidly develop, deploy, and rigorously evaluate CDS within an EHR. We also identified important considerations in applying these methods. This approach may be an important tool for improving the impact of and experience with CDS. TRIAL REGISTRATION: Flu alert trial: ClinicalTrials.gov NCT03415425; https://clinicaltrials.gov/ct2/show/NCT03415425. Tobacco alert trial: ClinicalTrials.gov NCT03714191; https://clinicaltrials.gov/ct2/show/NCT03714191.

Patient Interactions With an Automated Conversational Agent Delivering Pretest Genetics Education: Descriptive Study.

BACKGROUND: Cancer genetic testing to assess an individual's cancer risk and to enable genomics-informed cancer treatment has grown exponentially in the past decade. Because of this continued growth and a shortage of health care workers, there is a need for automated strategies that provide high-quality genetics services to patients to reduce the clinical demand for genetics providers. Conversational agents have shown promise in managing mental health, pain, and other chronic conditions and are increasingly being used in cancer genetic services. However, research on how patients interact with these agents to satisfy their information needs is limited. OBJECTIVE: Our primary aim is to assess user interactions with a conversational agent for pretest genetics education. METHODS: We conducted a feasibility study of user interactions with a conversational agent who delivers pretest genetics education to primary care patients without cancer who are eligible for cancer genetic evaluation. The conversational agent provided scripted content similar to that delivered in a pretest genetic counseling visit for cancer genetic testing. Outside of a core set of information delivered to all patients, users were able to navigate within the chat to request additional content in their areas of interest. An artificial intelligence-based preprogrammed library was also established to allow users to ask open-ended questions to the conversational agent. Transcripts of the interactions were recorded. Here, we describe the information selected, time spent to complete the chat, and use of the open-ended question feature. Descriptive statistics were used for quantitative measures, and thematic analyses were used for qualitative responses. RESULTS: We invited 103 patients to participate, of which 88.3% (91/103) were offered access to the conversational agent, 39% (36/91) started the chat, and 32% (30/91) completed the chat. Most users who completed the chat indicated that they wanted to continue with genetic testing (21/30, 70%), few were unsure (9/30, 30%), and no patient declined to move forward with testing. Those who decided to test spent an average of 10 (SD 2.57) minutes on the chat, selected an average of 1.87 (SD 1.2) additional pieces of information, and generally did not ask open-ended questions. Those who were unsure spent 4 more minutes on average (mean 14.1, SD 7.41; P=.03) on the chat, selected an average of 3.67 (SD 2.9) additional pieces of information, and asked at least one open-ended question. CONCLUSIONS: The pretest chat provided enough information for most patients to decide on cancer genetic testing, as indicated by the small number of open-ended questions. A subset of participants were still unsure about receiving genetic testing and may require additional education or interpersonal support before making a testing decision. Conversational agents have the potential to become a scalable alternative for pretest genetics education, reducing the clinical demand on genetics providers.

Impact of Clinical Decision Support on Antibiotic Prescribing for Acute Respiratory Infections: a Cluster Randomized Implementation Trial.

BACKGROUND: Clinical decision support (CDS) is a promising tool for reducing antibiotic prescribing for acute respiratory infections (ARIs). OBJECTIVE: To assess the impact of previously effective CDS on antibiotic-prescribing rates for ARIs when adapted and implemented in diverse primary care settings. DESIGN: Cluster randomized clinical trial (RCT) implementing a CDS tool designed to guide evidence-based evaluation and treatment of streptococcal pharyngitis and pneumonia. SETTING: Two large academic health system primary care networks with a mix of providers. PARTICIPANTS: All primary care practices within each health system were invited. All providers within participating clinic were considered a participant. Practices were randomized selection to a control or intervention group. INTERVENTIONS: Intervention practice providers had access to an integrated clinical prediction rule (iCPR) system designed to determine the risk of bacterial infection from reason for visit of sore throat, cough, or upper respiratory infection and guide evidence-based evaluation and treatment. MAIN OUTCOME(S): Change in overall antibiotic prescription rates. MEASURE(S): Frequency, rates, and type of antibiotics prescribed in intervention and controls groups. RESULTS: 33 primary care practices participated with 541 providers and 100,573 patient visits. Intervention providers completed the tool in 6.9% of eligible visits. Antibiotics were prescribed in 35% and 36% of intervention and control visits, respectively, showing no statistically significant difference. There were also no differences in rates of orders for rapid streptococcal tests (RR, 0.94; P = 0.11) or chest X-rays (RR, 1.01; P = 0.999) between groups. CONCLUSIONS: The iCPR tool was not effective in reducing antibiotic prescription rates for upper respiratory infections in diverse primary care settings. This has implications for the generalizability of CDS tools as they are adapted to heterogeneous clinical contexts. TRIAL REGISTRATION: Clinicaltrials.gov (NCT02534987). Registered August 26, 2015 at https://clinicaltrials.gov.

Implementation of a Behavioral Economics Electronic Health Record (BE-EHR) Module to Reduce Overtreatment of Diabetes in Older Adults.

BACKGROUND: Intensive glycemic control is of unclear benefit and carries increased risk for older adults with diabetes. The American Geriatrics Society's (AGS) Choosing Wisely (CW) guideline promotes less aggressive glycemic targets and reduction in pharmacologic therapy for older adults with type II diabetes. Meanwhile, behavioral economic (BE) approaches offer promise in influencing hard-to-change behavior, and previous studies have shown the benefits of using electronic health record (EHR) technology to encourage guideline adherence. OBJECTIVE: This study aimed to develop and pilot test an intervention that leverages BE with EHR technology to promote appropriate diabetes management in older adults. DESIGN: A pilot study within the New York University Langone Health (NYULH) EHR and Epic system to deliver BE-inspired nudges at five NYULH clinics at varying time points from July 12, 2018, through October 31, 2019. PARTICIPANTS: Clinicians across five practices in the NYULH system whose patients were older adults (age 76 and older) with type II diabetes. INTERVENTIONS: A BE-EHR module comprising six nudges was developed through a series of design workshops, interviews, user-testing sessions, and clinic visits. BE principles utilized in the nudges include framing, social norming, accountable justification, defaults, affirmation, and gamification. MAIN MEASURES: Patient-level CW compliance. KEY RESULTS: CW compliance increased 5.1% from a 16-week interval at baseline to a 16-week interval post intervention. From February 14 to June 5, 2018 (prior to the first nudge launch in Vanguard clinics), CW compliance for 1278 patients was mean (95% CI)-16.1% (14.1%, 18.1%). From July 3 to October 22, 2019 (after BE-EHR module launch at all five clinics), CW compliance for 680 patients was 21.2% (18.1%, 24.3%). CONCLUSIONS: The BE-EHR module shows promise for promoting the AGS CW guideline and improving diabetes management in older adults. A randomized controlled trial will commence to test the effectiveness of the intervention across 66 NYULH clinics. NIH TRIAL REGISTRY NUMBER: NCT03409523.

Development, Implementation, and Evaluation of a Personalized Machine Learning Algorithm for Clinical Decision Support: Case Study With Shingles Vaccination.

BACKGROUND: Although clinical decision support (CDS) alerts are effective reminders of best practices, their effectiveness is blunted by clinicians who fail to respond to an overabundance of inappropriate alerts. An electronic health record (EHR)-integrated machine learning (ML) algorithm is a potentially powerful tool to increase the signal-to-noise ratio of CDS alerts and positively impact the clinician's interaction with these alerts in general. OBJECTIVE: This study aimed to describe the development and implementation of an ML-based signal-to-noise optimization system (SmartCDS) to increase the signal of alerts by decreasing the volume of low-value herpes zoster (shingles) vaccination alerts. METHODS: We built and deployed SmartCDS, which builds personalized user activity profiles to suppress shingles vaccination alerts unlikely to yield a clinician's interaction. We extracted all records of shingles alerts from January 2017 to March 2019 from our EHR system, including 327,737 encounters, 780 providers, and 144,438 patients. RESULTS: During the 6 weeks of pilot deployment, the SmartCDS system suppressed an average of 43.67% (15,425/35,315) potential shingles alerts (appointments) and maintained stable counts of weekly shingles vaccination orders (326.3 with system active vs 331.3 in the control group; P=.38) and weekly user-alert interactions (1118.3 with system active vs 1166.3 in the control group; P=.20). CONCLUSIONS: All key statistics remained stable while the system was turned on. Although the results are promising, the characteristics of the system can be subject to future data shifts, which require automated logging and monitoring. We demonstrated that an automated, ML-based method and data architecture to suppress alerts are feasible without detriment to overall order rates. This work is the first alert suppression ML-based model deployed in practice and serves as foundational work in encounter-level customization of alert display to maximize effectiveness.

Design and implementation of a clinical decision support tool for primary palliative Care for Emergency Medicine (PRIM-ER).

BACKGROUND: The emergency department is a critical juncture in the trajectory of care of patients with serious, life-limiting illness. Implementation of a clinical decision support (CDS) tool automates identification of older adults who may benefit from palliative care instead of relying upon providers to identify such patients, thus improving quality of care by assisting providers with adhering to guidelines. The Primary Palliative Care for Emergency Medicine (PRIM-ER) study aims to optimize the use of the electronic health record by creating a CDS tool to identify high risk patients most likely to benefit from primary palliative care and provide point-of-care clinical recommendations. METHODS: A clinical decision support tool entitled Emergency Department Supportive Care Clinical Decision Support (Support-ED) was developed as part of an institutionally-sponsored value based medicine initiative at the Ronald O. Perelman Department of Emergency Medicine at NYU Langone Health. A multidisciplinary approach was used to develop Support-ED including: a scoping review of ED palliative care screening tools; launch of a workgroup to identify patient screening criteria and appropriate referral services; initial design and usability testing via the standard System Usability Scale questionnaire, education of the ED workforce on the Support-ED background, purpose and use, and; creation of a dashboard for monitoring and feedback. RESULTS: The scoping review identified the Palliative Care and Rapid Emergency Screening (P-CaRES) survey as a validated instrument in which to adapt and apply for the creation of the CDS tool. The multidisciplinary workshops identified two primary objectives of the CDS: to identify patients with indicators of serious life limiting illness, and to assist with referrals to services such as palliative care or social work. Additionally, the iterative design process yielded three specific patient scenarios that trigger a clinical alert to fire, including: 1) when an advance care planning document was present, 2) when a patient had a previous disposition to hospice, and 3) when historical and/or current clinical data points identify a serious life-limiting illness without an advance care planning document present. Monitoring and feedback indicated a need for several modifications to improve CDS functionality. CONCLUSIONS: CDS can be an effective tool in the implementation of primary palliative care quality improvement best practices. Health systems should thoughtfully consider tailoring their CDSs in order to adapt to their unique workflows and environments. The findings of this research can assist health systems in effectively integrating a primary palliative care CDS system seamlessly into their processes of care. TRIAL REGISTRATION: ClinicalTrials.gov Identifier: NCT03424109. Registered 6 February 2018, Grant Number: AT009844-01.

Integrating data from an online diabetes prevention program into an electronic health record and clinical workflow, a design phase usability study.

BACKGROUND: Health information is increasingly being digitally stored and exchanged. The public is regularly collecting and storing health-related data on their own electronic devices and in the cloud. Diabetes prevention is an increasingly important preventive health measure, and diet and exercise are key components of this. Patients are turning to online programs to help them lose weight. Despite primary care physicians being important in patients' weight loss success, there is no exchange of information between the primary care provider (PCP) and these online weight loss programs. There is an emerging opportunity to integrate this data directly into the electronic health record (EHR), but little is known about what information to share or how to share it most effectively. This study aims to characterize the preferences of providers concerning the integration of externally generated lifestyle modification data into a primary care EHR workflow. METHODS: We performed a qualitative study using two rounds of semi-structured interviews with primary care providers. We used an iterative design process involving primary care providers, health information technology software developers and health services researchers to develop the interface. RESULTS: Using grounded-theory thematic analysis 4 themes emerged from the interviews: 1) barriers to establishing healthy lifestyles, 2) features of a lifestyle modification program, 3) reporting of outcomes to the primary care provider, and 4) integration with primary care. These themes guided the rapid-cycle agile design process of an interface of data from an online diabetes prevention program into the primary care EHR workflow. CONCLUSIONS: The integration of external health-related data into the EHR must be embedded into the provider workflow in order to be useful to the provider and beneficial for the patient. Accomplishing this requires evaluation of that clinical workflow during software design. The development of this novel interface used rapid cycle iterative design, early involvement by providers, and usability testing methodology. This provides a framework for how to integrate external data into provider workflow in efficient and effective ways. There is now the potential to realize the importance of having this data available in the clinical setting for patient engagement and health outcomes.

Evidence-based risk communication: a systematic review.

BACKGROUND: Effective communication of risks and benefits to patients is critical for shared decision making. PURPOSE: To review the comparative effectiveness of methods of communicating probabilistic information to patients that maximize their cognitive and behavioral outcomes. DATA SOURCES: PubMed (1966 to March 2014) and CINAHL, EMBASE, and the Cochrane Central Register of Controlled Trials (1966 to December 2011) using several keywords and structured terms. STUDY SELECTION: Prospective or cross-sectional studies that recruited patients or healthy volunteers and compared any method of communicating probabilistic information with another method. DATA EXTRACTION: Two independent reviewers extracted study characteristics and assessed risk of bias. DATA SYNTHESIS: Eighty-four articles, representing 91 unique studies, evaluated various methods of numerical and visual risk display across several risk scenarios and with diverse outcome measures. Studies showed that visual aids (icon arrays and bar graphs) improved patients' understanding and satisfaction. Presentations including absolute risk reductions were better than those including relative risk reductions for maximizing accuracy and seemed less likely than presentations with relative risk reductions to influence decisions to accept therapy. The presentation of numbers needed to treat reduced understanding. Comparative effects of presentations of frequencies (such as 1 in 5) versus event rates (percentages, such as 20%) were inconclusive. LIMITATION: Most studies were small and highly variable in terms of setting, context, and methods of administering interventions. CONCLUSION: Visual aids and absolute risk formats can improve patients' understanding of probabilistic information, whereas numbers needed to treat can lessen their understanding. Due to study heterogeneity, the superiority of any single method for conveying probabilistic information is not established, but there are several good options to help clinicians communicate with patients. PRIMARY FUNDING SOURCE: None.

Longitudinal adoption rates of complex decision support tools in primary care.

Translating research findings into practice promises to standardise care. Translation includes the integration of evidence-based guidelines at the point of care, discerning the best methods to disseminate research findings and models to sustain the implementation of best practices.By applying usability testing to clinical decision support(CDS) design, overall adoption rates of 60% can be realised.What has not been examined is how long adoption rates are sustained and the characteristics associated with long-term use. We conducted secondary analysis to decipher the factors impacting sustained use of CD Stools. This study was a secondary data analysis from a clinical trial conducted at an academic institution in New York City. Study data was identified patients electronic health records (EHR). The trial was to test the implementation of an integrated clinical prediction rule(iCPR) into the EHR. The primary outcome variable was iCPR tool acceptance of the tool. iCPR tool completion and iCPR smartest completion were additional outcome variables of interest. The secondary aim was to examine user characteristics associated with iCPR tool use in later time periods. Characteristics of interest included age, resident year, use of electronic health records (yes/no) and use of best practice alerts (BPA) (yes/no). Generalised linear mixed models (GLiMM) were used to compare iCPR use over time for each outcome of interest: namely, iCPR acceptance, iCPR completion and iCPR smartset completion.GLiMM was also used to examine resident characteristics associated with iCPR tool use in later time periods; specifically, intermediate and long-term (ie, 90+days). The tool was accepted, on average, 82.18% in the first 90 days (short-term period). The use decreases to 56.07% and 45.61% in intermediate and long-term time periods, respectively. There was a significant association between iCPR tool completion and time periods(p<0.0001). There was no significant difference in iCPR tool completion between resident encounters in the intermediate and long-term periods (p<0.6627). There was a significant association between iCPR smartset completion and time periods (p<0.0021). There were no significant associations between iCPR smartest completion and any of the four predictors of interest. We examined the frequencies of components of the iCPR tool being accepted over time by individual clinicians. Rates of adoption of the different components of the tool decreased substantially over time. The data suggest that over time and prolonged exposure to CDS tools, providers are less likely to utilise the tool. It is not clear if it is fatigue with the CDS tool, acquired knowledge of the clinical prediction rule, or gained clinical experience and gestalt that are influencing adoption rates. Further analysis of individual adoption rates over time and the impact it has on clinical outcomes should be conducted.

Construction of the Digital Health Equity-Focused Implementation Research Conceptual Model - Bridging the Divide Between Equity-focused Digital Health and Implementation Research.

Digital health implementations and investments continue to expand. As the reliance on digital health increases, it is imperative to implement technologies with inclusive and accessible approaches. A conceptual model can be used to guide equity-focused digital health implementations to improve suitability and uptake in diverse populations. The objective of this study is expand an implementation model with recommendations on the equitable implementation of new digital health technologies. The Digital Health Equity-Focused Implementation Research (DH-EquIR) conceptual model was developed based on a rigorous review of digital health implementation and health equity literature. The Equity-Focused Implementation Research for Health Programs (EquIR) model was used as a starting point and merged with digital equity and digital health implementation models. Existing theoretical frameworks and models were appraised as well as individual equity-sensitive implementation studies. Patient and program-related concepts related to digital equity, digital health implementation, and assessment of social/digital determinants of health were included. Sixty-two articles were analyzed to inform the adaption of the EquIR model for digital health. These articles included digital health equity models and frameworks, digital health implementation models and frameworks, research articles, guidelines, and concept analyses. Concepts were organized into EquIR conceptual groupings, including population health status, planning the program, designing the program, implementing the program, and equity-focused implementation outcomes. The adapted DH-EquIR conceptual model diagram was created as well as detailed tables displaying related equity concepts, evidence gaps in source articles, and analysis of existing equity-related models and tools. The DH-EquIR model serves to guide digital health developers and implementation specialists to promote the inclusion of health-equity planning in every phase of implementation. In addition, it can assist researchers and product developers to avoid repeating the mistakes that have led to inequities in the implementation of digital health across populations.

Leveraging Machine Learning to Develop Digital Engagement Phenotypes of Users in a Digital Diabetes Prevention Program: Evaluation Study.

BACKGROUND: Digital diabetes prevention programs (dDPPs) are effective "digital prescriptions" but have high attrition rates and program noncompletion. To address this, we developed a personalized automatic messaging system (PAMS) that leverages SMS text messaging and data integration into clinical workflows to increase dDPP engagement via enhanced patient-provider communication. Preliminary data showed positive results. However, further investigation is needed to determine how to optimize the tailoring of support technology such as PAMS based on a user's preferences to boost their dDPP engagement. OBJECTIVE: This study evaluates leveraging machine learning (ML) to develop digital engagement phenotypes of dDPP users and assess ML's accuracy in predicting engagement with dDPP activities. This research will be used in a PAMS optimization process to improve PAMS personalization by incorporating engagement prediction and digital phenotyping. This study aims (1) to prove the feasibility of using dDPP user-collected data to build an ML model that predicts engagement and contributes to identifying digital engagement phenotypes, (2) to describe methods for developing ML models with dDPP data sets and present preliminary results, and (3) to present preliminary data on user profiling based on ML model outputs. METHODS: Using the gradient-boosted forest model, we predicted engagement in 4 dDPP individual activities (physical activity, lessons, social activity, and weigh-ins) and general activity (engagement in any activity) based on previous short- and long-term activity in the app. The area under the receiver operating characteristic curve, the area under the precision-recall curve, and the Brier score metrics determined the performance of the model. Shapley values reflected the feature importance of the models and determined what variables informed user profiling through latent profile analysis. RESULTS: We developed 2 models using weekly and daily DPP data sets (328,821 and 704,242 records, respectively), which yielded predictive accuracies above 90%. Although both models were highly accurate, the daily model better fitted our research plan because it predicted daily changes in individual activities, which was crucial for creating the "digital phenotypes." To better understand the variables contributing to the model predictor, we calculated the Shapley values for both models to identify the features with the highest contribution to model fit; engagement with any activity in the dDPP in the last 7 days had the most predictive power. We profiled users with latent profile analysis after 2 weeks of engagement (Bayesian information criterion=-3222.46) with the dDPP and identified 6 profiles of users, including those with high engagement, minimal engagement, and attrition. CONCLUSIONS: Preliminary results demonstrate that applying ML methods with predicting power is an acceptable mechanism to tailor and optimize messaging interventions to support patient engagement and adherence to digital prescriptions. The results enable future optimization of our existing messaging platform and expansion of this methodology to other clinical domains. TRIAL REGISTRATION: ClinicalTrials.gov NCT04773834; https://www.clinicaltrials.gov/ct2/show/NCT04773834. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.2196/26750.

Mixed methods assessment of the influence of demographics on medical advice of ChatGPT.

OBJECTIVES: To evaluate demographic biases in diagnostic accuracy and health advice between generative artificial intelligence (AI) (ChatGPT GPT-4) and traditional symptom checkers like WebMD. MATERIALS AND METHODS: Combination symptom and demographic vignettes were developed for 27 most common symptom complaints. Standardized prompts, written from a patient perspective, with varying demographic permutations of age, sex, and race/ethnicity were entered into ChatGPT (GPT-4) between July and August 2023. In total, 3 runs of 540 ChatGPT prompts were compared to the corresponding WebMD Symptom Checker output using a mixed-methods approach. In addition to diagnostic correctness, the associated text generated by ChatGPT was analyzed for readability (using Flesch-Kincaid Grade Level) and qualitative aspects like disclaimers and demographic tailoring. RESULTS: ChatGPT matched WebMD in 91% of diagnoses, with a 24% top diagnosis match rate. Diagnostic accuracy was not significantly different across demographic groups, including age, race/ethnicity, and sex. ChatGPT's urgent care recommendations and demographic tailoring were presented significantly more to 75-year-olds versus 25-year-olds (P < .01) but were not statistically different among race/ethnicity and sex groups. The GPT text was suitable for college students, with no significant demographic variability. DISCUSSION: The use of non-health-tailored generative AI, like ChatGPT, for simple symptom-checking functions provides comparable diagnostic accuracy to commercially available symptom checkers and does not demonstrate significant demographic bias in this setting. The text accompanying differential diagnoses, however, suggests demographic tailoring that could potentially introduce bias. CONCLUSION: These results highlight the need for continued rigorous evaluation of AI-driven medical platforms, focusing on demographic biases to ensure equitable care.

Quantifying the impact of telemedicine and patient medical advice request messages on physicians' work-outside-work.

The COVID-19 pandemic has boosted digital health utilization, raising concerns about increased physicians' after-hours clinical work ("work-outside-work"). The surge in patients' digital messages and additional time spent on work-outside-work by telemedicine providers underscores the need to evaluate the connection between digital health utilization and physicians' after-hours commitments. We examined the impact on physicians' workload from two types of digital demands - patients' messages requesting medical advice (PMARs) sent to physicians' inbox (inbasket), and telemedicine. Our study included 1716 ambulatory-care physicians in New York City regularly practicing between November 2022 and March 2023. Regression analyses assessed primary and interaction effects of (PMARs) and telemedicine on work-outside-work. The study revealed a significant effect of PMARs on physicians' work-outside-work and that this relationship is moderated by physicians' specialties. Non-primary care physicians or specialists experienced a more pronounced effect than their primary care peers. Analysis of their telemedicine load revealed that primary care physicians received fewer PMARs and spent less time in work-outside-work with more telemedicine. Specialists faced increased PMARs and did more work-outside-work as telemedicine visits increased which could be due to the difference in patient panels. Reducing PMAR volumes and efficient inbasket management strategies needed to reduce physicians' work-outside-work. Policymakers need to be cognizant of potential disruptions in physicians carefully balanced workload caused by the digital health services.