Large language models encode clinical knowledge.

Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA3, MedMCQA4, PubMedQA5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics6), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications.

Behavioral Health and the Comprehensive Primary Care (CPC) Initiative: findings from the 2014 CPC behavioral health survey.

BACKGROUND: Incorporating behavioral health care into patient centered medical homes is critical for improving patient health and care quality while reducing costs. Despite documented effectiveness of behavioral health integration (BHI) in primary care settings, implementation is limited outside of large health systems. We conducted a survey of BHI in primary care practices participating in the Comprehensive Primary Care (CPC) initiative, a four-year multi-payer initiative of the Centers for Medicare and Medicaid Services (CMS). We sought to explore associations between practice characteristics and the extent of BHI to illuminate possible factors influencing successful implementation. METHOD: We fielded a survey that addressed six substantive domains (integrated space, training, access, communication and coordination, treatment planning, and available resources) and five behavioral health conditions (depression, anxiety, pain, alcohol use disorder, and cognitive function). Descriptive statistics compared BHI survey respondents to all CPC practices, documented the availability of behavioral health providers, and primary care and behavioral health provider communication. Bivariate relationships compared provider and practice characteristics and domain scores. RESULTS: One hundred sixty-one of 188 eligible primary care practices completed the survey (86% response rate). Scores indicated basic to good baseline implementation of BHI in all domains, with lowest scores on communication and coordination and highest scores for depression. Higher scores were associated with: having any behavioral health provider, multispecialty practice, patient-centered medical home designation, and having any communication between behavioral health and primary care providers. CONCLUSIONS: This study provides useful data on opportunities and challenges of scaling BHI integration linked to primary care transformation. Payment reform models such as CPC can assist in BHI promotion and development.

Should the Affordable Care Act's preventive services coverage provision be used to widely disseminate whole genome sequencing to Americans?

I argue that the provision of the Patient Protection and Affordable Care Act (ACA) of 2010, which eliminates cost sharing for preventive services, should be utilized as a pathway for reimbursing whole genome sequencing (WGS) and making it widely available to most Americans. This act provides multiple routes for determining which preventive services receive this designation. Three of these routes should be considered as pathways for reimbursing WGS, including approval by the United States Preventive Services Task Force, inclusion in the guidelines of the American Academy of Pediatrics Bright Futures Project, and classification as a preventive service for women by the Institute of Medicine. There are valid arguments against the expansion of this technology, including inadequate national and state laws prohibiting genetic discrimination, informed consent limitations, and potentially expensive genome interpretations. These concerns should not inhibit the wide dissemination of this technology, as current efforts by the NIH and industry to expand the use of genome sequencing demonstrate. The ACA should be used as a tool to prevent disparities in access to genome information in the United States and avoid the development of a two-tiered health system based on those with and without genome sequence data.

Cardiologists' Perspectives on BiDil and the Use of Race in Drug Prescribing.

OBJECTIVES: We explored cardiologists' attitudes and prescribing patterns specific to the use of generic isosorbide dinitrate and hydralazine hydrochloride, and the fixed-dose patented drug, BiDil. BACKGROUND: Since the Food and Drug Administration approved BiDil in 2005 with an indication for self-identified black patients, disagreement about the appropriateness of race-based drugs has intensified and led to calls for providers and researchers to abandon race-based delimitations. This paper reports empirical evidence of cardiologists' views on BiDil's race-based indication and their ongoing inertia with respect to the debate about BiDil. METHODS: We conducted a 2010 cross-sectional online survey of members of the Association of Black Cardiologists. RESULTS: Fifty-nine cardiologists responded to the survey. Most participants (62.7%) prescribed BiDil to their patients. More than 40% of respondents did not prescribe BiDil to any non-African Americans. When considering whether to prescribe BiDil, a patient's race determined by physician assessment was the third most important factor considered by participants. The majority of participants (72.7%) selected symptoms as the most important factor. Most participants (59.2%) perceived race as defining biologically distinct individuals. Respondents prescribed BiDil more often to African American patients than non-African American patients. However, they prescribed the generic components that makeup BiDil to African Americans and non-African American patients similarly. CONCLUSIONS: The survey provides useful findings that, when viewed within the context of ongoing debates about race-based medicine, show little progress toward appropriately utilizing BiDil to maximize health outcomes, yet, might inform the development of practical and effective guidelines concerning the use of race in medicine.

Regional differences in awareness and attitudes regarding genetic testing for disease risk and ancestry.

Little is known about the lay public's awareness and attitudes concerning genetic testing and what factors influence their perspectives. The existing literature focuses mainly on ethnic and socioeconomic differences; however, here we focus on how awareness and attitudes regarding genetic testing differ by geographical regions in the US. We compared awareness and attitudes concerning genetic testing for disease risk and ancestry among 452 adults (41% Black and 67% female) in four major US cities, Norman, OK; Cincinnati, OH; Harlem, NY; and Washington, DC; prior to their participation in genetic ancestry testing. The OK participants reported more detail about their personal ancestries (p = 0.02) and valued ancestry testing over disease testing more than all other sites (p < 0.01). The NY participants were more likely than other sites to seek genetic testing for disease (p = 0.01) and to see benefit in finding out more about one's ancestry (p = 0.02), while the DC participants reported reading and hearing more about genetic testing for African ancestry than all other sites (p < 0.01). These site differences were not better accounted for by sex, age, education, self-reported ethnicity, religion, or previous experience with genetic testing/counseling. Regional differences in awareness and attitudes transcend traditional demographic predictors, such as ethnicity, age and education. Local sociocultural factors, more than ethnicity and socioeconomic status, may influence the public's awareness and belief systems, particularly with respect to genetics.

Compulsory service programmes for recruiting health workers in remote and rural areas: do they work?

Compulsory service programmes have been used worldwide as a way to deploy and retain a professional health workforce within countries. Other names for these programmes include "obligatory", "mandatory", "required" and "requisite." All these different programme names refer to a country's law or policy that governs the mandatory deployment and retention of a heath worker in the underserved and/or rural areas of the country for a certain period of time. This study identified three different types of compulsory service programmes in 70 countries. These programmes are all governed by some type of regulation, ranging from a parliamentary law to a policy within the ministry of health. Depending on the country, doctors, nurses, midwives and all types of professional allied health workers are required to participate in the programme. Some of the compliance-enforcement measures include withholding full registration until obligations are completed, withholding degree and salary, or imposing large fines. This paper aims to explain these programmes more clearly, to identify countries that have or had such programmes, to develop a typology for the different kinds and to discuss the programmes in the light of important issues that are related to policy concepts and implementation. As governments consider the cost of investment in health professionals' education, the loss of health professionals to emigration and the lack of health workers in many geographic areas, they are using compulsory service requirements as a way to deploy and retain the health workforce.

Teaching health policy to residents--three-year experience with a multi-specialty curriculum.

INTRODUCTION: Most residents have limited education or exposure to health policy during residency. AIMS: We developed a course to (1) educate residents on health policy topics applicable to daily physician practice; (2) expose residents to health policy careers through visits with policy makers and analysts; (3) promote personal engagement in health policy. SETTING: Residents registered for a 3-week elective offered twice annually through the George Washington University Department of Health Policy. PROGRAM DESCRIPTION: The course format includes: daily required readings and small-group seminars with policy experts, interactive on-site visits with policy makers, and final team presentations to senior faculty on topical health policy issues. PROGRAM EVALUATION: One hundred thirty residents from 14 specialties have completed the course to date. Seventy completed our post-course survey. Most participants [59 (84%)] felt the course was very or extremely helpful. Participant self-ratings increased from pre- to post-course in overall knowledge of health policy [2 (3%) good or excellent before, 58 (83%) after], likelihood of teaching policy concepts to peers [20 (25%) vs. 62 (86%)], and likelihood of pursuing further health policy training [28 (37%) vs. 56 (82%)]. CONCLUSIONS: This 3-week elective in health policy improves self-reported knowledge and interest in health policy research, advocacy, and teaching.

Performance on Electronic Clinical Quality Measures in the Comprehensive Primary Care Initiative.

Electronic clinical quality measures (eCQMs) that capture data from electronic health records promise accurate and timely measurement, but their use has been limited in payment. The Comprehensive Primary Care initiative sponsored by the Centers for Medicare & Medicaid Services used eCQMs as part of a shared savings incentive. To assess performance, the authors developed benchmarks for 11 measures and compared performance of initiative and benchmark practices. Initiative practices outperformed benchmark practices on 8 measures in 2015 (between 6.3 and 17.7 percentage points) and 9 measures in 2016 (between 1.7 and 20 percentage points). Initiative practices improved significantly on 7 measures from 2015 to 2016 (between 3.3 and 8.6 percentage points). For 3 measures, the improvement was greater than benchmark practices that reported the same measures in a 2-year period (between 1 and 8.9 percentage points). The authors conclude that eCQMs can be used for payment.

Uses and Limitations of Claims-based Performance Feedback Reports: Lessons From the Comprehensive Primary Care Initiative.

BACKGROUND: Performance feedback is central to data-driven models of quality improvement, but the use of claims-based data for feedback has received little attention. PURPOSE: To examine the challenges, uses, and limitations of quarterly Medicare claims-based performance feedback reports generated for practices participating in the Comprehensive Primary Care (CPC) initiative from 2012 to 2015. METHODS: Mixed methods study of nearly 500 CPC practices in seven regions, combining pilot testing; systematic monitoring; surveys; in-depth interviews; user feedback; and input from data feedback team. RESULTS: Designing reports required addressing issues about timing, data completeness and reliability, variations in patient risk across practices, and use of benchmarks and metrics understandable to users. Practices' ability to use reports constructively depended on their experience, analytic resources, expectations, and perceptions about the role of primary care in improving reported outcomes. CONCLUSIONS: Generating claims-based feedback reports that support practices' quality improvement efforts requires a significant investment of analytic expertise, time, resources, continuous improvement, and technical assistance. IMPLICATIONS: Claims-based performance feedback can provide insight into patterns of patients' care across provider settings and opportunities for improvement, but practices need data from other sources to manage patients in real time or assess the short-term effects of specific changes in care delivery.

The role of genetic and sociopolitical definitions of race in clinical trials.

Although the concept of race has been disputed for decades, race continues to be used as a variable in biomedical research. Public Law 103-43 calls on the National Institutes of Health to develop guidelines for defining "minority group" and "their subpopulations" for the purposes of ensuring that they are included in clinical trials. Current guidelines use census racial categories, even though these categories are labeled as not scientific by their creator, the Office of Management and Budget. Three policy options exist for improving the National Institutes of Health Policy on Reporting Race and Ethnicity: (1) using genetic ancestry instead of census racial categories;(2) developing a standardized definition of race using current science; and(3) redefining minority group populations and subpopulations using social environment variables rather than census racial categories.

Genetic and social environment interactions and their impact on health policy.

Genetic and social factors are not as separate as once thought. Researchers within the social sciences are beginning to realize that genetics and the social environment interact synergistically to affect health behaviors and outcomes. This way of thinking is leading to new research models and is influencing the development of research initiatives. The importance of this gene-social environment paradigm is evident in current and proposed health policies, and future research likely will spur further questions related to various areas of public policy.

Toward better Alzheimer's research information sources for the public.

The National Plan to Address Alzheimer's Disease calls for a new relationship between researchers and members of the public. This relationship is one that provides research information to patients and allows patients to provide ideas to researchers. One way to describe it is a "bidirectional translational relationship." Despite the numerous sources of online and offline information about Alzheimer's disease, there is no information source which currently provides this interaction. This article proposes the creation an Alzheimer's research information source dedicated to monitoring Alzheimer's research literature and providing user friendly, publicly accessible summaries of data written specifically for a lay audience. This information source should contain comprehensive, updated, user friendly, publicly available, reviews of Alzheimer's research and utilize existing online multimedia/social networking tools to provide information in useful formats that help patients, caregivers, and researchers learn rapidly from one another.

National culturally and linguistically appropriate services standards--mandates or not?

The U.S. Department of Health and Human Services' Office of Minority Health (OMH) promotes National Standards for Culturally and Linguistically Appropriate Services. According to the Office, four of these standards are "mandates" that are considered federal requirements for all recipients of federal funds. These standards focus on increasing language access to health care services for limited English proficient (LEP) people. Relying on this statement, multiple organizations, scholars and advocates promote these mandates in their communities. However, the DHHS Office of Inspector General stated in a 2010 report that these standards are only voluntary and not mandated. This contradiction deserves the attention of public health advocates for LEP people. This article seeks to clarify this contradiction so that public health advocates and experts can provide accurate information to people they serve and consider whether they should work with to develop truly mandatory language access standards as a condition of federal funding.