ABSTRACT
OBJECTIVE: Efforts to reduce documentation burden (DocBurden) for all health professionals (HP) are aligned with national initiatives to improve clinician wellness and patient safety. Yet DocBurden has not been precisely defined, limiting national conversations and rigorous, reproducible, and meaningful measures. Increasing attention to DocBurden motivated this work to establish a standard definition of DocBurden, with the emergence of excessive DocBurden as a term. METHODS: We conducted a scoping review of DocBurden definitions and descriptions, searching six databases for scholarly, peer-reviewed, and gray literature sources, using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extensions for Scoping Review (PRISMA-ScR) guidance. For the concept clarification phase of work, we used the American Nursing Informatics Association (ANIA)'s 6-Domains of Burden Framework. RESULTS: A total of 153 articles were included based on a priori criteria. Most articles described a focus on DocBurden, but only 18% (n=28) provided a definition. We define excessive DocBurden as the stress and unnecessarily heavy work a HP or healthcare team experiences when usability of documentation systems and documentation activities (i.e., generation, review, analysis and synthesis of patient data) are not aligned in support of care delivery. A negative connotation was attached to burden without a neutral state in included sources, which does not align with dictionary definitions of burden. CONCLUSIONS: Existing literature does not distinguish between a baseline or required task load to conduct patient care resulting from usability issues(DocBurden), and the unnecessarily heavy tasks and requirements that contribute to excessive DocBurden. Our definition of excessive DocBurden explicitly acknowledges this distinction, to support development of meaningful measures for understanding and intervening on excessive DocBurden locally, nationally and internationally.
ABSTRACT
Importance: The Sentinel System is a key component of the US Food and Drug Administration (FDA) postmarketing safety surveillance commitment and uses clinical health care data to conduct analyses to inform drug labeling and safety communications, FDA advisory committee meetings, and other regulatory decisions. However, observational data are frequently deemed insufficient for reliable evaluation of safety concerns owing to limitations in underlying data or methodology. Advances in large language models (LLMs) provide new opportunities to address some of these limitations. However, careful consideration is necessary for how and where LLMs can be effectively deployed for these purposes. Observations: LLMs may provide new avenues to support signal-identification activities to identify novel adverse event signals from narrative text of electronic health records. These algorithms may be used to support epidemiologic investigations examining the causal relationship between exposure to a medical product and an adverse event through development of probabilistic phenotyping of health outcomes of interest and extraction of information related to important confounding factors. LLMs may perform like traditional natural language processing tools by annotating text with controlled vocabularies with additional tailored training activities. LLMs offer opportunities for enhancing information extraction from adverse event reports, medical literature, and other biomedical knowledge sources. There are several challenges that must be considered when leveraging LLMs for postmarket surveillance. Prompt engineering is needed to ensure that LLM-extracted associations are accurate and specific. LLMs require extensive infrastructure to use, which many health care systems lack, and this can impact diversity, equity, and inclusion, and result in obscuring significant adverse event patterns in some populations. LLMs are known to generate nonfactual statements, which could lead to false positive signals and downstream evaluation activities by the FDA and other entities, incurring substantial cost. Conclusions and Relevance: LLMs represent a novel paradigm that may facilitate generation of information to support medical product postmarket surveillance activities that have not been possible. However, additional work is required to ensure LLMs can be used in a fair and equitable manner, minimize false positive findings, and support the necessary rigor of signal detection needed for regulatory activities.
Subject(s)
Natural Language Processing , Product Surveillance, Postmarketing , United States Food and Drug Administration , Product Surveillance, Postmarketing/methods , Humans , United States , Electronic Health RecordsABSTRACT
Suicide prevention requires risk identification, appropriate intervention, and follow-up. Traditional risk identification relies on patient self-reporting, support network reporting, or face-to-face screening with validated instruments or history and physical exam. In the last decade, statistical risk models have been studied and more recently deployed to augment clinical judgment. Models have generally been found to be low precision or problematic at scale due to low incidence. Few have been tested in clinical practice, and none have been tested in clinical trials to our knowledge. Methods: We report the results of a pragmatic randomized controlled trial (RCT) in three outpatient adult Neurology clinic settings. This two-arm trial compared the effectiveness of Interruptive and Non-Interruptive Clinical Decision Support (CDS) to prompt further screening of suicidal ideation for those predicted to be high risk using a real-time, validated statistical risk model of suicide attempt risk, with the decision to screen as the primary end point. Secondary outcomes included rates of suicidal ideation and attempts in both arms. Manual chart review of every trial encounter was used to determine if suicide risk assessment was subsequently documented. Results: From August 16, 2022, through February 16, 2023, our study randomized 596 patient encounters across 561 patients for providers to receive either Interruptive or Non-Interruptive CDS in a 1:1 ratio. Adjusting for provider cluster effects, Interruptive CDS led to significantly higher numbers of decisions to screen (42%=121/289 encounters) compared to Non-Interruptive CDS (4%=12/307) (odds ratio=17.7, p-value <0.001). Secondarily, no documented episodes of suicidal ideation or attempts occurred in either arm. While the proportion of documented assessments among those noting the decision to screen was higher for providers in the Non-Interruptive arm (92%=11/12) than in the Interruptive arm (52%=63/121), the interruptive CDS was associated with more frequent documentation of suicide risk assessment (63/289 encounters compared to 11/307, p-value<0.001). Conclusions: In this pragmatic RCT of real-time predictive CDS to guide suicide risk assessment, Interruptive CDS led to higher numbers of decisions to screen and documented suicide risk assessments. Well-powered large-scale trials randomizing this type of CDS compared to standard of care are indicated to measure effectiveness in reducing suicidal self-harm. ClinicalTrials.gov Identifier: NCT05312437.
ABSTRACT
OBJECTIVE: Large-language models (LLMs) can potentially revolutionize health care delivery and research, but risk propagating existing biases or introducing new ones. In epilepsy, social determinants of health are associated with disparities in care access, but their impact on seizure outcomes among those with access remains unclear. Here we (1) evaluated our validated, epilepsy-specific LLM for intrinsic bias, and (2) used LLM-extracted seizure outcomes to determine if different demographic groups have different seizure outcomes. MATERIALS AND METHODS: We tested our LLM for differences and equivalences in prediction accuracy and confidence across demographic groups defined by race, ethnicity, sex, income, and health insurance, using manually annotated notes. Next, we used LLM-classified seizure freedom at each office visit to test for demographic outcome disparities, using univariable and multivariable analyses. RESULTS: We analyzed 84 675 clinic visits from 25 612 unique patients seen at our epilepsy center. We found little evidence of bias in the prediction accuracy or confidence of outcome classifications across demographic groups. Multivariable analysis indicated worse seizure outcomes for female patients (OR 1.33, P ≤ .001), those with public insurance (OR 1.53, P ≤ .001), and those from lower-income zip codes (OR ≥1.22, P ≤ .007). Black patients had worse outcomes than White patients in univariable but not multivariable analysis (OR 1.03, P = .66). CONCLUSION: We found little evidence that our LLM was intrinsically biased against any demographic group. Seizure freedom extracted by LLM revealed disparities in seizure outcomes across several demographic groups. These findings quantify the critical need to reduce disparities in the care of people with epilepsy.
Subject(s)
Epilepsy , Healthcare Disparities , Seizures , Humans , Female , Male , Adult , Middle Aged , Natural Language Processing , Social Determinants of Health , Adolescent , Young Adult , LanguageABSTRACT
This Viewpoint posits that to improve public understanding of the system, the Vaccine Adverse Event Reporting System (VAERS) could use a more accurate name, well-defined guidance about the reporting system's nature and use, and comprehensible information about an event's verification status.
Subject(s)
Adverse Drug Reaction Reporting Systems , Communication , Vaccines , United States , Vaccines/adverse effectsABSTRACT
BACKGROUND: Electronic health records (EHRs) present navigation challenges due to time-consuming searches across segmented data. Voice assistants can improve clinical workflows by allowing natural language queries and contextually aware navigation of the EHR. OBJECTIVES: To develop a voice-mediated EHR assistant and interview providers to inform its future refinement. METHODS: The Vanderbilt EHR Voice Assistant (VEVA) was developed as a responsive web application and designed to accept voice inputs and execute the appropriate EHR commands. Fourteen providers from Vanderbilt Medical Center were recruited to participate in interactions with VEVA and to share their experience with the technology. The purpose was to evaluate VEVA's overall usability, gather qualitative feedback, and detail suggestions for enhancing its performance. RESULTS: VEVA's mean system usability scale score was 81 based on the 14 providers' evaluations, which was above the standard 50th percentile score of 68. For all five summaries evaluated (overview summary, A1C results, blood pressure, weight, and health maintenance), most providers offered a positive review of VEVA. Several providers suggested modifications to make the technology more useful in their practice, ranging from summarizing current medications to changing VEVA's speech rate. Eight of the providers (64%) reported they would be willing to use VEVA in its current form. CONCLUSION: Our EHR voice assistant technology was deemed usable by most providers. With further improvements, voice assistant tools such as VEVA have the potential to improve workflows and serve as a useful adjunct tool in health care.
Subject(s)
Electronic Health Records , Software , Language , TechnologyABSTRACT
OBJECTIVES: Automated phenotyping algorithms can reduce development time and operator dependence compared to manually developed algorithms. One such approach, PheNorm, has performed well for identifying chronic health conditions, but its performance for acute conditions is largely unknown. Herein, we implement and evaluate PheNorm applied to symptomatic COVID-19 disease to investigate its potential feasibility for rapid phenotyping of acute health conditions. MATERIALS AND METHODS: PheNorm is a general-purpose automated approach to creating computable phenotype algorithms based on natural language processing, machine learning, and (low cost) silver-standard training labels. We applied PheNorm to cohorts of potential COVID-19 patients from 2 institutions and used gold-standard manual chart review data to investigate the impact on performance of alternative feature engineering options and implementing externally trained models without local retraining. RESULTS: Models at each institution achieved AUC, sensitivity, and positive predictive value of 0.853, 0.879, 0.851 and 0.804, 0.976, and 0.885, respectively, at quantiles of model-predicted risk that maximize F1. We report performance metrics for all combinations of silver labels, feature engineering options, and models trained internally versus externally. DISCUSSION: Phenotyping algorithms developed using PheNorm performed well at both institutions. Performance varied with different silver-standard labels and feature engineering options. Models developed locally at one site also worked well when implemented externally at the other site. CONCLUSION: PheNorm models successfully identified an acute health condition, symptomatic COVID-19. The simplicity of the PheNorm approach allows it to be applied at multiple study sites with substantially reduced overhead compared to traditional approaches.
Subject(s)
Algorithms , COVID-19 , Humans , Electronic Health Records , Machine Learning , Natural Language ProcessingABSTRACT
Post marketing safety surveillance depends in part on the ability to detect concerning clinical events at scale. Spontaneous reporting might be an effective component of safety surveillance, but it requires awareness and understanding among healthcare professionals to achieve its potential. Reliance on readily available structured data such as diagnostic codes risk under-coding and imprecision. Clinical textual data might bridge these gaps, and natural language processing (NLP) has been shown to aid in scalable phenotyping across healthcare records in multiple clinical domains. In this study, we developed and validated a novel incident phenotyping approach using unstructured clinical textual data agnostic to Electronic Health Record (EHR) and note type. It's based on a published, validated approach (PheRe) used to ascertain social determinants of health and suicidality across entire healthcare records. To demonstrate generalizability, we validated this approach on two separate phenotypes that share common challenges with respect to accurate ascertainment: 1) suicide attempt; 2) sleep-related behaviors. With samples of 89,428 records and 35,863 records for suicide attempt and sleep-related behaviors, respectively, we conducted silver standard (diagnostic coding) and gold standard (manual chart review) validation. We showed Area Under the Precision-Recall Curve of â¼ 0.77 (95% CI 0.75-0.78) for suicide attempt and AUPR â¼ 0.31 (95% CI 0.28-0.34) for sleep-related behaviors. We also evaluated performance by coded race and demonstrated differences in performance by race were dissimilar across phenotypes and require algorithmovigilance and debiasing prior to implementation.
ABSTRACT
Objective: Large-language models (LLMs) in healthcare have the potential to propagate existing biases or introduce new ones. For people with epilepsy, social determinants of health are associated with disparities in access to care, but their impact on seizure outcomes among those with access to specialty care remains unclear. Here we (1) evaluated our validated, epilepsy-specific LLM for intrinsic bias, and (2) used LLM-extracted seizure outcomes to test the hypothesis that different demographic groups have different seizure outcomes. Methods: First, we tested our LLM for intrinsic bias in the form of differential performance in demographic groups by race, ethnicity, sex, income, and health insurance in manually annotated notes. Next, we used LLM-classified seizure freedom at each office visit to test for outcome disparities in the same demographic groups, using univariable and multivariable analyses. Results: We analyzed 84,675 clinic visits from 25,612 patients seen at our epilepsy center 2005-2022. We found no differences in the accuracy, or positive or negative class balance of outcome classifications across demographic groups. Multivariable analysis indicated worse seizure outcomes for female patients (OR 1.33, p = 3×10-8), those with public insurance (OR 1.53, p = 2×10-13), and those from lower-income zip codes (OR ≥ 1.22, p ≤ 6.6×10-3). Black patients had worse outcomes than White patients in univariable but not multivariable analysis (OR 1.03, p = 0.66). Significance: We found no evidence that our LLM was intrinsically biased against any demographic group. Seizure freedom extracted by LLM revealed disparities in seizure outcomes across several demographic groups. These findings highlight the critical need to reduce disparities in the care of people with epilepsy.
ABSTRACT
Advances in artificial intelligence (AI) have been changing the landscape in daily life and the practice of medicine. As these tools have evolved to become consumer-friendly, AI has become more accessible to many individuals, including applicants to medical school. With the rise of AI models capable of generating complex passages of text, questions have arisen regarding the appropriateness of using such tools to assist in the preparation of medical school applications. In this commentary, the authors offer a brief history of AI tools in medicine and describe large language models, a form of AI capable of generating natural language text passages. They question whether AI assistance should be considered inappropriate in preparing applications and compare it with the assistance some applicants receive from family, physician friends, or consultants. They call for clearer guidelines on what forms of assistance-human and technological-are permitted in the preparation of medical school applications. They recommend that medical schools steer away from blanket bans on AI tools in medical education and instead consider mechanisms for knowledge sharing about AI between students and faculty members, incorporation of AI tools into assignments, and the development of curricula to teach the use of AI tools as a competency.
Subject(s)
Artificial Intelligence , Education, Medical , Humans , Schools, Medical , Curriculum , FacultyABSTRACT
Fine-grained organic-rich sediments (FGORS) are accumulating in estuaries worldwide, with multi-faceted negative ecosystem impacts. A pilot experiment was carried out in a residential canal of the Indian River Lagoon estuary (IRL, Florida, USA) using an aeration treatment intended to mitigate the harmful ecological effects of organic-rich sediment pollution. Planktonic and benthic communities were monitored, and environmental data collected throughout the aeration process. Results were compared against control conditions to evaluate the efficacy of aeration in the mitigation of FGORS. During the aeration process, hurricane Irma impacted the study area, bringing heavy rainfall and spawning a brown tide event (Aureoumbra lagunensis). The overall thickness and volume of FGORS, and the organic content of surface sediments did not change during the aeration treatment. Dissolved oxygen was higher and ammonium concentrations were lower in aeration canal bottom water compared to the control canal. During treatment, aeration did facilitate benthic animal life when temperatures dropped below 25°C, likely due to water column mixing and the increased capacity of water to hold dissolved gasses. In general, aeration did not significantly change the planktonic community composition relative to the control canal, but, during the post-bloom period, aeration helped to weaken the brown tide and phytoplankton densities were 35-50% lower for A. lagunensis in aeration canal surface water compared to the control canal. Aeration has important management applications and may be useful for mitigating algal blooms in flow-restricted areas and promoting benthic communities in cooler environments.
Subject(s)
Estuaries , Phytoplankton , Animals , Ecosystem , Eutrophication , Plankton , Water , Geologic Sediments , Environmental MonitoringABSTRACT
Clinical informatics remains underappreciated among medical students in part due to a lack of integration into undergraduate medical education (UME). New developments in the study and practice of medicine are traditionally introduced via formal integration into undergraduate medical curricula. While this path has certain advantages, curricular changes are slow and may fail to showcase the breadth of clinical informatics activities. Less formal and more flexible approaches can circumvent these drawbacks. Interest groups (IGs), which are organized through the Association of American Medical College Careers in Medicine (CiM) program, exemplify the informal approach. CiM IGs are student-led groups that provide exposure to different specialty options, acting as an adjunct to the traditional medical curriculum. While the primary purpose of these groups is to assist students applying to residency programs, we took a novel approach of using an IG to increase student exposure to an area of medicine that had not yet been formally integrated at our institution. IGs provide unique advantages to formal integration into a curriculum as they can be more easily setup and can quickly respond to student interests. Furthermore, IGs can act synergistically with UME, acting as proving grounds for ideas that can lead to new courses. We believe that the lessons and takeaways from our experience can act as a guide for those interested in starting similar organizations at their own schools.
Subject(s)
Education, Medical, Undergraduate , Medical Informatics , Physicians , Humans , Public Opinion , Curriculum , Medical Informatics/educationABSTRACT
Harmful algal blooms (HABs) may quickly travel and inoculate new water bodies via currents and runoff in estuaries. The role of in-situ prokaryotic communities in the re-establishment and growth of inoculated algal blooms remains unknown. A novel on-board incubation experiment was employed to simulate the sudden surge of algal blooms to new estuarine waters and reveal possible outcomes. A dinoflagellate (Amphidinium carterae) and a diatom species (Thalassiosira weissflogii) which had bloomed in the Pearl River Estuary (PRE) area were cultured to bloom densities and reintroduced back into PRE natural seawaters. The diatom showed better adaptation ability to the new environment and increased significantly after the incubation. Simultaneously, particle-attached (PA) prokaryotic community structure was strongly influenced by adding of the diatom, with some opportunistic prokaryotes significantly enhanced in the diatom treatment. Whereas the dinoflagellate population did not increase following incubation, and their PA prokaryotic community showed no significant differences relative to the control. Metagenomic analyzes revealed that labile carbohydrates and organic nitrogen produced by the diatom contributed to the surge of certain PA prokaryotes. Genomic properties of a bacteria strain, which is affiliated with genus GMD16E07 (Planctomycetaceae) and comprised up to 50% of PA prokaryotes in the diatom treatment, was described here for the first time. Notably, the association of Planctomycetaceae and T. weissflogii likely represents symbiotic mutualism, with the diatom providing organic matter for Planctomycetaceae and the bacteria supplying vitamins and detoxifying nitriles and hydrogen peroxides in exchange. Therefore, the close association between Planctomycetaceae and T. weissflogii promoted the growth of both populations, and eventually facilitated the diatom bloom establishment.
Subject(s)
Diatoms , Dinoflagellida , Bacteria/genetics , Estuaries , Harmful Algal Bloom , RiversABSTRACT
BACKGROUND: The widespread adoption of electronic health records and a simultaneous increase in regulatory demands have led to an acceleration of documentation requirements among clinicians. The corresponding burden from documentation requirements is a central contributor to clinician burnout and can lead to an increased risk of suboptimal patient care. OBJECTIVE: To address the problem of documentation burden, the 25 by 5: Symposium to Reduce Documentation Burden on United States Clinicians by 75% by 2025 (Symposium) was organized to provide a forum for experts to discuss the current state of documentation burden and to identify specific actions aimed at dramatically reducing documentation burden for clinicians. METHODS: The Symposium consisted of six weekly sessions with 33 presentations. The first four sessions included panel presentations discussing the challenges related to documentation burden. The final two sessions consisted of breakout groups aimed at engaging attendees in establishing interventions for reducing clinical documentation burden. Steering Committee members analyzed notes from each breakout group to develop a list of action items. RESULTS: The Steering Committee synthesized and prioritized 82 action items into Calls to Action among three stakeholder groups: Providers and Health Systems, Vendors, and Policy and Advocacy Groups. Action items were then categorized into as short-, medium-, or long-term goals. Themes that emerged from the breakout groups' notes include the following: accountability, evidence is critical, education and training, innovation of technology, and other miscellaneous goals (e.g., vendors will improve shared knowledge databases). CONCLUSION: The Symposium successfully generated a list of interventions for short-, medium-, and long-term timeframes as a launching point to address documentation burden in explicit action-oriented ways. Addressing interventions to reduce undue documentation burden placed on clinicians will necessitate collaboration among all stakeholders.
Subject(s)
Burnout, Professional , Documentation , Burnout, Psychological , Electronic Health Records , Humans , Research Report , United StatesABSTRACT
OBJECTIVE: We describe the Clickbusters initiative implemented at Vanderbilt University Medical Center (VUMC), which was designed to improve safety and quality and reduce burnout through the optimization of clinical decision support (CDS) alerts. MATERIALS AND METHODS: We developed a 10-step Clickbusting process and implemented a program that included a curriculum, CDS alert inventory, oversight process, and gamification. We carried out two 3-month rounds of the Clickbusters program at VUMC. We completed descriptive analyses of the changes made to alerts during the process, and of alert firing rates before and after the program. RESULTS: Prior to Clickbusters, VUMC had 419 CDS alerts in production, with 488 425 firings (42 982 interruptive) each week. After 2 rounds, the Clickbusters program resulted in detailed, comprehensive reviews of 84 CDS alerts and reduced the number of weekly alert firings by more than 70 000 (15.43%). In addition to the direct improvements in CDS, the initiative also increased user engagement and involvement in CDS. CONCLUSIONS: At VUMC, the Clickbusters program was successful in optimizing CDS alerts by reducing alert firings and resulting clicks. The program also involved more users in the process of evaluating and improving CDS and helped build a culture of continuous evaluation and improvement of clinical content in the electronic health record.
Subject(s)
Decision Support Systems, Clinical , Medical Order Entry Systems , Electronic Health Records , HumansABSTRACT
PURPOSE: The purpose of this study was to evaluate the safety and efficacy of intra-articular injections of autologous peripheral blood stem cells (PBSCs) plus hyaluronic acid (HA) after arthroscopic subchondral drilling into massive chondral defects of the knee joint and to determine whether PBSC therapy can improve functional outcome and reduce pain of the knee joint better than HA plus physiotherapy. METHODS: This is a dual-center randomized controlled trial (RCT). Sixty-nine patients aged 18 to 55 years with International Cartilage Repair Society grade 3 and 4 chondral lesions (size ≥3 cm2) of the knee joint were randomized equally into (1) a control group receiving intra-articular injections of HA plus physiotherapy and (2) an intervention group receiving arthroscopic subchondral drilling into chondral defects and postoperative intra-articular injections of PBSCs plus HA. The coprimary efficacy endpoints were subjective International Knee Documentation Committee (IKDC) and Knee Injury and Osteoarthritis Outcome Score (KOOS)-pain subdomain measured at month 24. The secondary efficacy endpoints included all other KOOS subdomains, Numeric Rating Scale (NRS), and Magnetic Resonance Observation of Cartilage Repair Tissue (MOCART) scores. RESULTS: At 24 months, the mean IKDC scores for the control and intervention groups were 48.1 and 65.6, respectively (P < .0001). The mean for KOOS-pain subdomain scores were 59.0 (control) and 86.0 (intervention) with P < .0001. All other KOOS subdomain, NRS, and MOCART scores were statistically significant (P < .0001) at month 24. Moreover, for the intervention group, 70.8% of patients had IKDC and KOOS-pain subdomain scores exceeding the minimal clinically important difference values, indicating clinical significance. There were no notable adverse events that were unexpected and related to the study drug or procedures. CONCLUSIONS: Arthroscopic marrow stimulation with subchondral drilling into massive chondral defects of the knee joint followed by postoperative intra-articular injections of autologous PBSCs plus HA is safe and showed a significant improvement of clinical and radiologic scores compared with HA plus physiotherapy. LEVEL OF EVIDENCE: Level I, RCT.
Subject(s)
Arthroplasty, Subchondral , Cartilage, Articular , Peripheral Blood Stem Cells , Cartilage, Articular/surgery , Follow-Up Studies , Humans , Hyaluronic Acid , Knee Joint/surgery , Physical Therapy ModalitiesABSTRACT
Open discussions of social justice and health inequities may be an uncommon focus within information technology science, business, and health care delivery partnerships. However, the COVID-19 pandemic-which disproportionately affected Black, indigenous, and people of color-has reinforced the need to examine and define roles that technology partners should play to lead anti-racism efforts through our work. In our perspective piece, we describe the imperative to prioritize TechQuity-equity and social justice as a technology business strategy-through collaborating in partnerships that focus on eliminating racial and social inequities.
Subject(s)
COVID-19 , Racism , Humans , Pandemics , SARS-CoV-2 , TechnologyABSTRACT
OBJECTIVE: The goals of this study are to describe the value and impact of Project HealthDesign (PHD), a program of the Robert Wood Johnson Foundation that applied design thinking to personal health records, and to explore the applicability of the PHD model to another challenging translational informatics problem: the integration of AI into the healthcare system. MATERIALS AND METHODS: We assessed PHD's impact and value in 2 ways. First, we analyzed publication impact by calculating a PHD h-index and characterizing the professional domains of citing journals. Next, we surveyed and interviewed PHD grantees, expert consultants, and codirectors to assess the program's components and the potential future application of design thinking to artificial intelligence (AI) integration into healthcare. RESULTS: There was a total of 1171 unique citations to PHD-funded work (collective h-index of 25). Studies citing PHD span medical, legal, and computational journals. Participants stated that this project transformed their thinking, altered their career trajectory, and resulted in technology transfer into the commercial sector. Participants felt, in general, that the approach would be valuable in solving contemporary challenges integrating AI in healthcare including complex social questions, integrating knowledge from multiple domains, implementation, and governance. CONCLUSION: Design thinking is a systematic approach to problem-solving characterized by cooperation and collaboration. PHD generated significant impacts as measured by citations, reach, and overall effect on participants. PHD's design thinking methods are potentially useful to other work on cyber-physical systems, such as the use of AI in healthcare, to propose structural or policy-related changes that may affect adoption, value, and improvement of the care delivery system.