Driving Improvement in Health and Health Care: Setting Metrics for Medicaid Outcomes.

Clinical performance metrics are the foundation of the design and ultimate performance of North Carolina's Medicaid reform plan. This commentary describes the general approach of the state's Department of Health and Human Services in setting metrics, including goals, assumptions, and starting principles.

Bringing Patients Into the Patient-Centered Medical Home: Lessons Learned in a Large Primary Care Practice.

There is consensus that patients need to be engaged with their care, but how to do this in a primary care setting remains unclear. This case study demonstrates Patient Advisory Council engagement with the operations of a patient-centered medical home.

Innovations in the education of health professionals.

Dramatic and unprecedented changes in health care have altered the health care landscape and have significant implications for health professions education. This issue of the NCMJ explores these changes and highlights innovative models across the health professions that are designed to prepare graduates to practice in the emerging health care system and to deliver high-quality care in a cost-effective manner. These new educational programs--which include training for future doctors, nurses, dentists, pharmacists, and various allied health professionals--aim to prepare providers to meet the needs of North Carolina communities, and they use new educational models to give graduates the competencies they need to practice in health care teams and to contribute in other ways to improved health outcomes for the people of the state.

COVID-19 Impact on Family Medicine Residents Exam Performance.

BACKGROUND AND OBJECTIVES: The COVID-19 pandemic began interrupting family medicine residency training in spring 2020. While a decline in scores on the American Board of Family Medicine In-Training Examination (ITE) has been observed, whether this decline has translated into the high-stakes Family Medicine Certification Examination (FMCE) is unclear. The goal of this study was to systematically assess the magnitude of COVID-19 impact on medical knowledge acquisition during residency, as measured by the ITE and FMCE. METHODS: A total of 19,101 initial certification candidates from 2017 to 2022 were included in this study. Annual ITE scores and FMCE scores were reported on the same scale (200-800) and served as the outcome measure. We conducted multilevel regression analysis to determine ITE score growth and FMCE scores compared to cohorts prior to COVID-19. RESULTS: During COVID-19, the increase in ITE scores from postgraduate year 2 (PGY-2) to PGY-3 was 25.5 points less, representing a 57.6% relative decrease; and from PGY-3 ITE to FMCE, it was 8.6 points less, a 12.7% relative decrease, compared with cohorts prior to COVID-19. FMCE scores were 6.6 points less during COVID-19, representing a 1.2% relative decline from the average FMCE score prior to COVID-19. CONCLUSIONS: This study found nonsubstantive COVID-19 impact on FMCE scores, but a considerable knowledge acquisition decline during residency, especially during the PGY-2 to PGY-3 period. While COVID-19 impacted learning, our findings indicated that residencies were largely able to remediate knowledge deficits before residents took the FMCE.

Differences in Physician Performance and Self-rated Confidence on High- and Low-Stakes Knowledge Assessments in Board Certification.

INTRODUCTION: Evidence links assessment to optimal learning, affirming that physicians are more likely to study, learn, and practice skills when some form of consequence ("stakes") may result from an assessment. We lack evidence, however, on how physicians' confidence in their knowledge relates to performance on assessments, and whether this varies based on the stakes of the assessment. METHODS: Our retrospective repeated-measures design compared differences in patterns of physician answer accuracy and answer confidence among physicians participating in both a high-stakes and a low-stakes longitudinal assessment of the American Board of Family Medicine. RESULTS: After 1 and 2 years, participants were more often correct but less confident in their accuracy on a higher-stakes longitudinal knowledge assessment compared with a lower-stakes assessment. There were no differences in question difficulty between the two platforms. Variation existed between platforms in time spent answering questions, use of resources to answer questions, and perceived question relevance to practice. DISCUSSION: This novel study of physician certification suggests that the accuracy of physician performance increases with higher stakes, even as self-reported confidence in their knowledge declines. It suggests that physicians may be more engaged in higher-stakes compared with lower-stakes assessments. With medical knowledge growing exponentially, these analyses provide an example of the complementary roles of higher- and lower-stakes knowledge assessment in supporting physician learning during continuing specialty board certification.

Natural history of practice transformation: development and initial testing of an outcomes-based model.

PURPOSE: Practice transformation is the cornerstone of the future of family medicine and health care reform, but little is known about how the process occurs. We sought to develop and test a model of the natural history of practice transformation. METHODS: We developed an outcomes-based model of how a practice moves through practice transformation in 2 phases: (1) initial model created through meetings with collaborative experts and practice facilitators, and (2) clinical and practice systems change reports examined from the first group of participating North Carolina Improving Performance In Practice practices to test and further refine the model. RESULTS: The resultant model described motivators and supports to transformation. Three emerging practice patterns were identified with the model: transformed practices experiencing robust improvement, activated practices with moderate change, and engaged practices with minimal change in measured quality over a 2-year period. Transformed practices showed broad-based improvement; some reached a threshold and others continued to improve. These practices had highly engaged leadership and used data to drive decisions. Activated practices had a slower improvement trajectory, usually encountering a barrier that took time to overcome (eg, extracting population data, spreading practice changes). Engaged practices did not improve or were unable to sustain change; despite good intentions, multiple competing distractions interfered with practice transformation. CONCLUSIONS: Practice transformation is a continuous and long-term process. Internal and external practice motivations and specific practice supports provided by a community-based quality improvement program appear to have an impact on engagement, rate of quality improvement, and long-term sustainability. Early successes play a key role as practices learn how to change their performance.

Family physicians in the maternity care workforce: factors influencing declining trends.

Family physicians provide access to maternity care for a disproportionate share of rural and urban underserved communities. This paper aims to determine trends in maternity care provision by family physicians and the characteristics of family physicians that provide maternity care. We used American Board of Family Medicine survey data collected from every family physician during application for the Maintenance of Certification Examination to determine the percentage of family physicians that provided maternity care from 2000 to 2010. Using a cross-sectional study design, logistic regression analysis was performed to examine association between maternity care provision and various physician demographic and practice characteristics. Maternity care provision by family physicians declined from 23.3 % in 2000 to 9.7 % in 2010 (p < 0.0001). Family physicians who were female, younger and US medical graduates were more likely to practice maternity care. Practicing in a rural setting (OR = 2.2; 95 % CL 2.1-2.4), an educational setting (OR = 6.4; 95 % CL 5.7-7.1) and in either the Midwest (OR = 2.6; 95 % CL 2.3-2.9) or West (OR = 2.3; 95 % CL 2.1-2.6) were the strongest predictors of higher likelihood of providing maternity care. While family physicians continue to play an important role in providing maternity care in many parts of the United States, the steep decline in the percentage of family physicians providing maternity care is concerning. Formal collaborations with midwives and obstetrician-gynecologists, malpractice reform, payment changes and graduate medical education innovations are potential avenues to explore to ensure access to maternity care.

Medication documentation in a primary care network serving North Carolina medicaid patients: results of a cross-sectional chart review.

BACKGROUND: Medical records that do not accurately reflect the patient's current medication list are an open invitation to errors and may compromise patient safety. METHODS: This cross-sectional study compares primary care provider (PCP) medication lists and pharmacy claims for 100 patients seen in 8 primary care practices and examines the association of congruence with demographic, clinical, and practice characteristics. Medication list congruence was measured as agreement of pharmacy claims with the entire PCP chart, including current medication list, visit notes, and correspondence sections. RESULTS: Congruence between pharmacy claims and the PCP chart was 65%. Congruence was associated with large chronic disease burden, frequent PCP visits, group practice, and patient age ≥45 years. CONCLUSION: Agreement of medication lists between the PCP chart and pharmacy records is low. Medication documentation was more accurate among patients who have more chronic conditions, those who have frequent PCP visits, those whose practice has multiple providers, and those at least 45 years of age. Improved congruence among patients with multiple chronic conditions and in group practices may reflect more frequent visits and reviews by providers.

The American Board of Family Medicine's 8 Years of Experience with Differential Item Functioning.

INTRODUCTION: Differential item functioning (DIF) procedures flag examination questions in which examinees from different subpopulations who are of equal ability do not have the same probability of answering it correctly. Few medical certification boards employ DIF procedures because they do not collect the needed data on the examinee's race or ethnicity. This article summarizes the American Board of Family Medicine's (ABFM) combined use of DIF procedures and an expert panel to review certification questions for bias. METHODS: ABFM certification examination data from 2013 to 2020 were analyzed using a DIF procedure to flag questions with possible ethnic or racial bias. The flagged questions were reviewed by a racially and ethnically diverse panel of content experts. If the panel judged the source of the DIF was not clinically relevant for the practice of family medicine, the question was removed from the examination. RESULTS: Out of the 3487 questions analyzed, 374 unique questions (11%) were flagged by DIF procedures as potentially biased. Of the flagged questions, the review panel felt 4 should be removed for fairness. DISCUSSION: Using DIF procedures and panel review can improve the quality of the board certification questions and demonstrate the organization's commitment to avoid racial or ethnic bias.

Racial/Ethnic Representation Among American Board of Family Medicine Certification Candidates from 1970 to 2020.

BACKGROUND: Because improved patient outcomes and experiences have been associated with health care workforce diversity, efforts to create a diverse family physician workforce have increased. However, a metric that could properly measure family physician representation in various contexts has seldom been studied. OBJECTIVE: The goal of this study is to propose a new metric logRQ and use it to examine the diversification progress of American Board of Family Medicine (ABFM) certification candidates relative to national, state, and historic populations, as well as medical school matriculants. METHODS: We obtained race/ethnicity for certification candidates from the 2014 to 2020 ABFM Certification Examination Registration questionnaire and examined racial/ethnic representation relative to various populations via logRQs. RESULTS: The total sample comprised 26,368 initial certification candidates and 55,347 continuing certification candidates. Asian, Hispanic, and Black's logRQ increased by 0.51, 0.42, and 0.41, respectively, in initial certification candidates compared with continuing certification candidates. In addition, logRQ standard deviation ranged from 0.19 to 0.87 across States, indicating state-level variation. Although Black and Hispanic remained underrepresented, the degree of underrepresentation had improved substantially across the past 5 decades, with logRQ increasing from -2.12 (Black) and -1.16 (Hispanic) in the 1970s to -0.46 (Black) and -0.68 (Hispanic) in the 2010s. The race/ethnicity logRQs of 2020 initial certification candidates relative to 2013 to 2014 medical school matriculants were all near 0, reflecting equitable representation. CONCLUSION: We utilized the proposed metric logRQ to quantify the advancement in representation among ABFM certification candidates in different contexts. The proposed logRQ may serve as a useful tool to monitor representation progress systematically.