A toolbox for surfacing health equity harms and biases in large language models.

Large language models (LLMs) hold promise to serve complex health information needs but also have the potential to introduce harm and exacerbate health disparities. Reliably evaluating equity-related model failures is a critical step toward developing systems that promote health equity. We present resources and methodologies for surfacing biases with potential to precipitate equity-related harms in long-form, LLM-generated answers to medical questions and conduct a large-scale empirical case study with the Med-PaLM 2 LLM. Our contributions include a multifactorial framework for human assessment of LLM-generated answers for biases and EquityMedQA, a collection of seven datasets enriched for adversarial queries. Both our human assessment framework and our dataset design process are grounded in an iterative participatory approach and review of Med-PaLM 2 answers. Through our empirical study, we find that our approach surfaces biases that may be missed by narrower evaluation approaches. Our experience underscores the importance of using diverse assessment methodologies and involving raters of varying backgrounds and expertise. While our approach is not sufficient to holistically assess whether the deployment of an artificial intelligence (AI) system promotes equitable health outcomes, we hope that it can be leveraged and built upon toward a shared goal of LLMs that promote accessible and equitable healthcare.

Prospective Study of Non-Contrast, Abbreviated MRI for Hepatocellular Carcinoma Surveillance in Patients with Suboptimal Hepatic Visualisation on Ultrasound.

BACKGROUND: Biannual ultrasound (US) is recommended for hepatocellular carcinoma (HCC) surveillance in patients with cirrhosis. However, US has limited sensitivity for early-stage HCC, particularly in overweight cohorts, where hepatic visualisation is often inadequate. Currently there are no robust imaging surveillance strategies in patients with inadequate US visualisation. We investigated the ability of non-contrast, abbreviated magnetic resonance imaging (aMRI) to adequately visualise the liver for HCC surveillance in patients with previously inadequate US. METHODS: Patients undergoing US surveillance, where liver visualisation was inadequate (LI-RADS VIS-B and VIS-C), were prospectively recruited. Patients underwent non-contrast T2-weighted and diffusion-weighted aMRI. The images were reviewed and reported by an expert liver radiologist. Three independent, blinded radiologists assessed the aMRI visualisation quality using a binary score assessing five parameters (parenchymal definition, vascular definition, coverage of the liver, uniformity of liver appearance and signal-to-noise ratio). RESULTS: Thirty patients completed the aMRI protocol. The majority (90%) had underlying cirrhosis and were overweight (93.3%), with 50% obese and 20% severely obese. A total of 93.3% of the aMRI scans were of satisfactory quality. Six patients (20%) had hepatic abnormalities detected with aMRI that were not seen on their US: one HCC, one haemangioma and three clinically insignificant lesions. For the aMRI visualisation quality assessment, the coverage of the liver, vascular definition and parenchymal definition were consistently rated to be of sufficient quality by all three radiologists. CONCLUSIONS: Non-contrast aMRI provided good visualisation of the liver and detection of abnormalities in patients with inadequate US. aMRI should be further explored in a larger, prospective study as an alternative surveillance strategy in patients with inadequate US.

Impact of antiretroviral therapy during primary HIV infection on T-cell immunity after treatment interruption.

This study aims to understand the impact of early antiretroviral therapy (ART) on HIV-specific T-cell responses measured after treatment interruption may inform strategies to deliver ART-free immune-mediated viral suppression. HIV-specific T-cell immunity was analysed using gamma interferon enzyme-linked immunospot assays in two studies. SPARTAC included individuals with primary HIV infection randomised to 48 weeks of ART (n = 24) or no immediate therapy (n = 37). The PITCH (n = 7) cohort started antiretroviral therapy in primary infection for at least one year, followed by TI. In SPARTAC, participants treated in PHI for 48 weeks followed by TI for 12 weeks, and those who remained untreated for 60 weeks made similar HIV Gag-directed responses (both magnitude and breadth) at week 60. However, the treated group made a greater proportion of novel HIV Gag-directed responses by Week 60, suggestive of a greater reserve to produce new potentially protective responses. In the more intensively followed PITCH study, 6/7 participants showed dominant Gag and/or Pol-specific responses post-TI compared with pre-TI. Although early ART in PHI was not associated with major differences in HIV-specific immunity following TI compared with untreated participants, the potential to make more new Gag-directed responses warrants further investigation as this may inform strategies to achieve ART-free control.

Healthcare interactions prior to first hospital admission with alcohol-related liver disease.

BACKGROUND AND AIMS: To examine the healthcare contacts of patients in the year before an index admission to hospital with alcohol-related liver disease (ArLD) to identify where opportunities for earlier identification of alcohol use disorders (AUD) and ArLD and intervention may occur. METHODS: A retrospective cohort study using the regional database encompassing NHS organisations across North West London (344 general practitioner [GP] practices, 4 acute hospital trusts and 2 mental health and community health trusts). Patients who had an index admission with ArLD were identified through healthcare coding and compared with a control cohort. Healthcare contacts, blood tests and AUD testing in the year preceding admission were measured. RESULTS: The ArLD cohort had 1494 participants with an index hospital admission with ArLD. The control cohort included 4462 participants. In the year preceding an index admission with ArLD, 91% of participants had at least one contact with primary care with an average of 2.97 (SD 2.45) contacts; 80% (n = 1199/1494) attended ED, 68% attended an outpatient clinic, and 42% (n = 628/1494) had at least one inpatient admission. Only 9% of the ArLD (137/1494) had formal testing for AUD. Abnormal bilirubin and platelets were more common in the ArLD than the control cohort 25% (138/560) and 28% (231/837), respectively, v 1% (12/1228) and 1% (20/1784). CONCLUSIONS: Prior to an index admission with ArLD patients have numerous interactions with all healthcare settings, indicating missed opportunities for early identification and treatment.

Health equity assessment of machine learning performance (HEAL): a framework and dermatology AI model case study.

Background: Artificial intelligence (AI) has repeatedly been shown to encode historical inequities in healthcare. We aimed to develop a framework to quantitatively assess the performance equity of health AI technologies and to illustrate its utility via a case study. Methods: Here, we propose a methodology to assess whether health AI technologies prioritise performance for patient populations experiencing worse outcomes, that is complementary to existing fairness metrics. We developed the Health Equity Assessment of machine Learning performance (HEAL) framework designed to quantitatively assess the performance equity of health AI technologies via a four-step interdisciplinary process to understand and quantify domain-specific criteria, and the resulting HEAL metric. As an illustrative case study (analysis conducted between October 2022 and January 2023), we applied the HEAL framework to a dermatology AI model. A set of 5420 teledermatology cases (store-and-forward cases from patients of 20 years or older, submitted from primary care providers in the USA and skin cancer clinics in Australia), enriched for diversity in age, sex and race/ethnicity, was used to retrospectively evaluate the AI model's HEAL metric, defined as the likelihood that the AI model performs better for subpopulations with worse average health outcomes as compared to others. The likelihood that AI performance was anticorrelated to pre-existing health outcomes was estimated using bootstrap methods as the probability that the negated Spearman's rank correlation coefficient (i.e., "R") was greater than zero. Positive values of R suggest that subpopulations with poorer health outcomes have better AI model performance. Thus, the HEAL metric, defined as p (R >0), measures how likely the AI technology is to prioritise performance for subpopulations with worse average health outcomes as compared to others (presented as a percentage below). Health outcomes were quantified as disability-adjusted life years (DALYs) when grouping by sex and age, and years of life lost (YLLs) when grouping by race/ethnicity. AI performance was measured as top-3 agreement with the reference diagnosis from a panel of 3 dermatologists per case. Findings: Across all dermatologic conditions, the HEAL metric was 80.5% for prioritizing AI performance of racial/ethnic subpopulations based on YLLs, and 92.1% and 0.0% respectively for prioritizing AI performance of sex and age subpopulations based on DALYs. Certain dermatologic conditions were significantly associated with greater AI model performance compared to a reference category of less common conditions. For skin cancer conditions, the HEAL metric was 73.8% for prioritizing AI performance of age subpopulations based on DALYs. Interpretation: Analysis using the proposed HEAL framework showed that the dermatology AI model prioritised performance for race/ethnicity, sex (all conditions) and age (cancer conditions) subpopulations with respect to pre-existing health disparities. More work is needed to investigate ways of promoting equitable AI performance across age for non-cancer conditions and to better understand how AI models can contribute towards improving equity in health outcomes. Funding: Google LLC.

Using generative AI to investigate medical imagery models and datasets.

BACKGROUND: AI models have shown promise in performing many medical imaging tasks. However, our ability to explain what signals these models have learned is severely lacking. Explanations are needed in order to increase the trust of doctors in AI-based models, especially in domains where AI prediction capabilities surpass those of humans. Moreover, such explanations could enable novel scientific discovery by uncovering signals in the data that aren't yet known to experts. METHODS: In this paper, we present a workflow for generating hypotheses to understand which visual signals in images are correlated with a classification model's predictions for a given task. This approach leverages an automatic visual explanation algorithm followed by interdisciplinary expert review. We propose the following 4 steps: (i) Train a classifier to perform a given task to assess whether the imagery indeed contains signals relevant to the task; (ii) Train a StyleGAN-based image generator with an architecture that enables guidance by the classifier ("StylEx"); (iii) Automatically detect, extract, and visualize the top visual attributes that the classifier is sensitive towards. For visualization, we independently modify each of these attributes to generate counterfactual visualizations for a set of images (i.e., what the image would look like with the attribute increased or decreased); (iv) Formulate hypotheses for the underlying mechanisms, to stimulate future research. Specifically, present the discovered attributes and corresponding counterfactual visualizations to an interdisciplinary panel of experts so that hypotheses can account for social and structural determinants of health (e.g., whether the attributes correspond to known patho-physiological or socio-cultural phenomena, or could be novel discoveries). FINDINGS: To demonstrate the broad applicability of our approach, we present results on eight prediction tasks across three medical imaging modalities-retinal fundus photographs, external eye photographs, and chest radiographs. We showcase examples where many of the automatically-learned attributes clearly capture clinically known features (e.g., types of cataract, enlarged heart), and demonstrate automatically-learned confounders that arise from factors beyond physiological mechanisms (e.g., chest X-ray underexposure is correlated with the classifier predicting abnormality, and eye makeup is correlated with the classifier predicting low hemoglobin levels). We further show that our method reveals a number of physiologically plausible, previously-unknown attributes based on the literature (e.g., differences in the fundus associated with self-reported sex, which were previously unknown). INTERPRETATION: Our approach enables hypotheses generation via attribute visualizations and has the potential to enable researchers to better understand, improve their assessment, and extract new knowledge from AI-based models, as well as debug and design better datasets. Though not designed to infer causality, importantly, we highlight that attributes generated by our framework can capture phenomena beyond physiology or pathophysiology, reflecting the real world nature of healthcare delivery and socio-cultural factors, and hence interdisciplinary perspectives are critical in these investigations. Finally, we will release code to help researchers train their own StylEx models and analyze their predictive tasks of interest, and use the methodology presented in this paper for responsible interpretation of the revealed attributes. FUNDING: Google.

An intentional approach to managing bias in general purpose embedding models.

Advances in machine learning for health care have brought concerns about bias from the research community; specifically, the introduction, perpetuation, or exacerbation of care disparities. Reinforcing these concerns is the finding that medical images often reveal signals about sensitive attributes in ways that are hard to pinpoint by both algorithms and people. This finding raises a question about how to best design general purpose pretrained embeddings (GPPEs, defined as embeddings meant to support a broad array of use cases) for building downstream models that are free from particular types of bias. The downstream model should be carefully evaluated for bias, and audited and improved as appropriate. However, in our view, well intentioned attempts to prevent the upstream components-GPPEs-from learning sensitive attributes can have unintended consequences on the downstream models. Despite producing a veneer of technical neutrality, the resultant end-to-end system might still be biased or poorly performing. We present reasons, by building on previously published data, to support the reasoning that GPPEs should ideally contain as much information as the original data contain, and highlight the perils of trying to remove sensitive attributes from a GPPE. We also emphasise that downstream prediction models trained for specific tasks and settings, whether developed using GPPEs or not, should be carefully designed and evaluated to avoid bias that makes models vulnerable to issues such as distributional shift. These evaluations should be done by a diverse team, including social scientists, on a diverse cohort representing the full breadth of the patient population for which the final model is intended.

Rapid antiretroviral therapy in primary HIV-1 infection enhances immune recovery.

OBJECTIVE: We present findings from a large cohort of individuals treated during primary HIV infection (PHI) and examine the impact of time from HIV-1 acquisition to antiretroviral therapy (ART) initiation on clinical outcomes. We also examine the temporal changes in the demographics of individuals presenting with PHI to inform HIV-1 prevention strategies. METHODS: Individuals who fulfilled the criteria of PHI and started ART within 3 months of confirmed HIV-1 diagnosis were enrolled between 2009 and 2020. Baseline demographics of those diagnosed between 2009 and 2015 (before preexposure prophylaxis (PrEP) and universal ART availability) and 2015-2020 (post-PrEP and universal ART availability) were compared. We examined the factors associated with immune recovery and time to viral suppression. RESULTS: Two hundred four individuals enrolled, 144 from 2009 to 2015 and 90 from 2015 to 2020; median follow-up was 33 months. At PHI, the median age was 33 years; 4% were women, 39% were UK-born, and 84% were MSM. The proportion of UK-born individuals was 47% in 2009-2015, compared with 29% in 2015-2020. There was an association between earlier ART initiation after PHI diagnosis and increased immune recovery; each day that ART was delayed was associated with a lower likelihood of achieving a CD4 + cell count more than 900 cells/µl [hazard ratio 0.99 (95% confidence interval, 95% CI 0.98-0.99), P  = 0.02) and CD4/CD8 more than 1.0 (hazard ratio 0.98 (95% CI 0.97-0.99). CONCLUSION: Early initiation of ART at PHI diagnosis is associated with enhanced immune recovery, providing further evidence to support immediate ART in the context of PHI. Non-UK-born MSM accounts for an increasing proportion of those with primary infection; UK HIV-1 prevention strategies should better target this group.

The value of standards for health datasets in artificial intelligence-based applications.

Artificial intelligence as a medical device is increasingly being applied to healthcare for diagnosis, risk stratification and resource allocation. However, a growing body of evidence has highlighted the risk of algorithmic bias, which may perpetuate existing health inequity. This problem arises in part because of systemic inequalities in dataset curation, unequal opportunity to participate in research and inequalities of access. This study aims to explore existing standards, frameworks and best practices for ensuring adequate data diversity in health datasets. Exploring the body of existing literature and expert views is an important step towards the development of consensus-based guidelines. The study comprises two parts: a systematic review of existing standards, frameworks and best practices for healthcare datasets; and a survey and thematic analysis of stakeholder views of bias, health equity and best practices for artificial intelligence as a medical device. We found that the need for dataset diversity was well described in literature, and experts generally favored the development of a robust set of guidelines, but there were mixed views about how these could be implemented practically. The outputs of this study will be used to inform the development of standards for transparency of data diversity in health datasets (the STANDING Together initiative).

Large language models encode clinical knowledge.

Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA3, MedMCQA4, PubMedQA5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics6), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications.

Top ten priorities for global saltmarsh restoration, conservation and ecosystem service research.

Coastal saltmarshes provide globally important ecosystem services including 'blue carbon' sequestration, flood protection, pollutant remediation, habitat provision and cultural value. Large portions of marshes have been lost or fragmented as a result of land reclamation, embankment construction, and pollution. Sea level rise threatens marsh survival by blocking landward migration where coastlines have been developed. Research-informed saltmarsh conservation and restoration efforts are helping to prevent further loss, yet significant knowledge gaps remain. Using a mixed methods approach, this paper identifies ten research priorities through an online questionnaire and a residential workshop attended by an international, multi-disciplinary network of 35 saltmarsh experts spanning natural, physical and social sciences across research, policy, and practitioner sectors. Priorities have been grouped under four thematic areas of research: Saltmarsh Area Extent, Change and Restoration Potential (including past, present, global variation), Spatio-social contexts of Ecosystem Service delivery (e.g. influences of environmental context, climate change, and stakeholder groups on service provisioning), Patterns and Processes in saltmarsh functioning (global drivers of saltmarsh ecosystem structure/function) and Management and Policy Needs (how management varies contextually; challenges/opportunities for management). Although not intended to be exhaustive, the challenges, opportunities, and strategies for addressing each research priority examined here, providing a blueprint of the work that needs to be done to protect saltmarshes for future generations.

A large outbreak of COVID-19 in a UK prison, October 2020 to April 2021.

Prisons are susceptible to outbreaks. Control measures focusing on isolation and cohorting negatively affect wellbeing. We present an outbreak of coronavirus disease 2019 (COVID-19) in a large male prison in Wales, UK, October 2020 to April 2021, and discuss control measures.We gathered case-information, including demographics, staff-residence postcode, resident cell number, work areas/dates, test results, staff interview dates/notes and resident prison-transfer dates. Epidemiological curves were mapped by prison location. Control measures included isolation (exclusion from work or cell-isolation), cohorting (new admissions and work-area groups), asymptomatic testing (case-finding), removal of communal dining and movement restrictions. Facemask use and enhanced hygiene were already in place. Whole-genome sequencing (WGS) and interviews determined the genetic relationship between cases plausibility of transmission.Of 453 cases, 53% (n = 242) were staff, most aged 25-34 years (11.5% females, 27.15% males) and symptomatic (64%). Crude attack-rate was higher in staff (29%, 95% CI 26-64%) than in residents (12%, 95% CI 9-15%).Whole-genome sequencing can help differentiate multiple introductions from person-to-person transmission in prisons. It should be introduced alongside asymptomatic testing as soon as possible to control prison outbreaks. Timely epidemiological investigation, including data visualisation, allowed dynamic risk assessment and proportionate control measures, minimising the reduction in resident welfare.

Acamprosate may be safer than baclofen for the treatment of alcohol use disorder in patients with cirrhosis: a first description of use in real-world clinical practice.

OBJECTIVES: Patients with alcohol use disorder (AUD) and liver cirrhosis benefit from stopping alcohol intake. Baclofen has been trialled for AUD in cirrhosis and appears to be effective. However, in patients without cirrhosis acamprosate is safer and more efficacious. Acamprosate is rarely used in cirrhosis due to safety concerns: the only published report was for 24 h in a controlled setting. Our centre uses both medications off-label in cirrhotic patients. We performed an audit to pragmatically compare the safety of acamprosate to baclofen in these patients. METHODS: The electronic records of patients prescribed acamprosate or baclofen between 01/04/17 and 31/03/20 were retrospectively reviewed. Adverse events and abstinence at last follow-up were compared by Student's t-test, Mann-Whitney U or chi-square test. Confounding variables were evaluated by logistic regression. RESULTS: In total 48 cirrhotic patients taking acamprosate (median 84 days, range 2-524); 44 baclofen (247 days, 8-910) met inclusion criteria. At baseline, 41% had Childs-Pugh B or C cirrhosis. More patients taking baclofen had an unplanned hospital admission or attendance (23 vs 13; P = 0.013) and the mean number per patient was higher (1.6 vs 0.6; P = 0.032). Sub-group analysis revealed increased admissions in actively drinking patients prescribed baclofen to achieve abstinence (mean 2.4 vs 0.6; P = 0.020); acamprosate use was associated with a reduced chance of admission or attendance (OR, 0.284; 0.095-0.854; P = 0.025) independent of treatment length. No difference in efficacy was observed. CONCLUSIONS: In patients with cirrhosis, acamprosate was associated with fewer unplanned admissions than baclofen, hence may be safer despite historical concerns.

SARS-CoV-2 sero-prevalence in the workforces of three large workplaces in South Wales: a sero-epidemiological study.

BACKGROUND: Sero-prevalence studies quantify the proportion of a population that has antibodies against SARS-CoV-2, and can be used to identify the extent of the COVID-19 pandemic at a population level. The aim of the study was to assess the sero-prevalence of SARS-CoV-2 antibodies in the workforce at three workplaces: a food factory, non-food factory and call-centre. METHODS: Nine hundred ninety-three participants were recruited from three workplaces in South Wales. Participants completed a questionnaire and had a lateral flow point-of-care SARS-CoV-2 antibody test administered by a healthcare professional. The data were analysed using multivariable logistic regression, both using complete records only and following multiple imputation. RESULTS: The sero-prevalence of SARS-CoV-2 antibodies ranged from 4% (n = 17/402) in the non-food factory to 10% (n = 28/281) in the food factory (OR 2.93; 95% CI 1.26 to 6.81). After taking account of confounding factors evidence of a difference remained (cOR comparing food factory to call centre (2.93; 95% CI 1.26 to 6.81) and non-food factory (3.99; 95% CI 1.97 to 8.08) respectively). The SARS-CoV-2 antibody prevalence also varied between roles within workplaces. People working in office based roles had a 2.23 times greater conditional odds (95% CI 1.02 to 4.87) of being positive for SARS-CoV-2 antibodies than those working on the factory floor. CONCLUSION: The sero-prevalence of SARS-CoV-2 antibodies varied by workplace and work role. Whilst it is not possible to state whether these differences are due to COVID-19 transmission within the workplaces, it highlights the importance of considering COVID-19 transmission in a range of workplaces and work roles.

Conventional histological and cytological staining with simultaneous immunohistochemistry enabled by invisible chromogens.

Conventional histological stains, such as hematoxylin plus eosin (H&E), and immunohistochemistry (IHC) are mainstays of histology that provide complementary diagnostic information. H&E and IHC currently require separate slides, because the stains would otherwise obscure one another. This consumes small specimen, limiting the total amount of testing. Additionally, performing H&E and IHC on different slides does not permit comparison of staining at the single cell level, since the same cells are not present on each slide, and alignment of tissue features can be problematic due to changes in tissue landscape with sectioning. We have solved these problems by performing conventional staining and IHC on the same slide using invisible IHC chromogens, such that the chromogens are not visible when viewing the conventional stain and the conventional stain is excluded from images of the IHC. Covalently deposited chromogens provided a convenient route to invisible chromogen design and are stable to reagents used in conventional staining. A dual-camera brightfield microscope system was developed that permits simultaneous viewing of both visible conventional stains and invisible IHC chromogens. Simultaneous staining was demonstrated on several formalin-fixed paraffin-embedded tissue specimens using single and duplex IHC, with chromogens that absorb ultraviolet and near infrared light, followed by H&E staining. The concept was extended to other conventional stains, including mucicarmine special stain and Papanicoulou stain, and further extended to cytology specimens. In addition to interactive video review, images were recorded using multispectral imaging and image processing to provide flexible production of color composite images and enable quantitative analysis.

Behavior science in the evolving world of digital health: considerations on anticipated opportunities and challenges.

Digital health promises to increase intervention reach and effectiveness for a range of behavioral health outcomes. Behavioral scientists have a unique opportunity to infuse their expertise in all phases of a digital health intervention, from design to implementation. The aim of this study was to assess behavioral scientists' interests and needs with respect to digital health endeavors, as well as gather expert insight into the role of behavioral science in the evolution of digital health. The study used a two-phased approach: (a) a survey of behavioral scientists' current needs and interests with respect to digital health endeavors (n = 346); (b) a series of interviews with digital health stakeholders for their expert insight on the evolution of the health field (n = 15). In terms of current needs and interests, the large majority of surveyed behavioral scientists (77%) already participate in digital health projects, and from those who have not done so yet, the majority (65%) reported intending to do so in the future. In terms of the expected evolution of the digital health field, interviewed stakeholders anticipated a number of changes, from overall landscape changes through evolving models of reimbursement to more significant oversight and regulations. These findings provide a timely insight into behavioral scientists' current needs, barriers, and attitudes toward the use of technology in health care and public health. Results might also highlight the areas where behavioral scientists can leverage their expertise to both enhance digital health's potential to improve health, as well as to prevent the potential unintended consequences that can emerge from scaling the use of technology in health care.