Machine Learning to Identify Clusters in Family Medicine Diplomate Motivations and Their Relationship to Continuing Certification Exam Outcomes: Findings and Potential Future Implications.

BACKGROUND: The potential for machine learning (ML) to enhance the efficiency of medical specialty boards has not been explored. We applied unsupervised ML to identify archetypes among American Board of Family Medicine (ABFM) Diplomates regarding their practice characteristics and motivations for participating in continuing certification, then examined associations between motivation patterns and key recertification outcomes. METHODS: Diplomates responding to the 2017 to 2021 ABFM Family Medicine continuing certification examination surveys selected motivations for choosing to continue certification. We used Chi-squared tests to examine difference proportions of Diplomates failing their first recertification examination attempt who endorsed different motivations for maintaining certification. Unsupervised ML techniques were applied to generate clusters of physicians with similar practice characteristics and motivations for recertifying. Controlling for physician demographic variables, we used logistic regression to examine the effect of motivation clusters on recertification examination success and validated the ML clusters by comparison with a previously created classification schema developed by experts. RESULTS: ML clusters largely recapitulated the intrinsic/extrinsic framework devised by experts previously. However, the identified clusters achieved a more equal partitioning of Diplomates into homogenous groups. In both ML and human clusters, physicians with mainly extrinsic or mixed motivations had lower rates of examination failure than those who were intrinsically motivated. DISCUSSION: This study demonstrates the feasibility of using ML to supplement and enhance human interpretation of board certification data. We discuss implications of this demonstration study for the interaction between specialty boards and physician Diplomates.

Differences in Physician Performance and Self-rated Confidence on High- and Low-Stakes Knowledge Assessments in Board Certification.

INTRODUCTION: Evidence links assessment to optimal learning, affirming that physicians are more likely to study, learn, and practice skills when some form of consequence ("stakes") may result from an assessment. We lack evidence, however, on how physicians' confidence in their knowledge relates to performance on assessments, and whether this varies based on the stakes of the assessment. METHODS: Our retrospective repeated-measures design compared differences in patterns of physician answer accuracy and answer confidence among physicians participating in both a high-stakes and a low-stakes longitudinal assessment of the American Board of Family Medicine. RESULTS: After 1 and 2 years, participants were more often correct but less confident in their accuracy on a higher-stakes longitudinal knowledge assessment compared with a lower-stakes assessment. There were no differences in question difficulty between the two platforms. Variation existed between platforms in time spent answering questions, use of resources to answer questions, and perceived question relevance to practice. DISCUSSION: This novel study of physician certification suggests that the accuracy of physician performance increases with higher stakes, even as self-reported confidence in their knowledge declines. It suggests that physicians may be more engaged in higher-stakes compared with lower-stakes assessments. With medical knowledge growing exponentially, these analyses provide an example of the complementary roles of higher- and lower-stakes knowledge assessment in supporting physician learning during continuing specialty board certification.

To Effectively Address Complex Healthcare Problems, Continuing Professional Development Must Evolve.

ABSTRACT: Continuing professional development aims to provide health professionals with the knowledge, skills, and competencies needed to improve care. Physicians and other clinicians increasingly practice within complex health care delivery organizations aiming to improve the care of populations of patients with multiple problems and differing needs. These organizations are composed of local units in different departments and venues; these teams and the patients they care for change over time. Improving outcomes within constantly changing complex organizations delivering population care takes time and persistence. It takes time to equip critical masses of clinicians and other personnel with knowledge and skills to effect change. Although some changes might be simple, those involving new workflows require implementation support. Not all change will be smooth; individuals need opportunities to learn from and adjust their early intervention efforts, measure effectiveness of change, and sustain successful practices. Longitudinal support is necessary to affect change over complex organizations. This essay proposes that to be more supportive and valuable to health care delivery organizations, continuing professional development needs to intentionally participate in longitudinal, collaborative, context-specific, team-based interventions. An expanded menu of evaluation approaches will better describe the role of continuing professional development in helping health care professionals and organizations address increasingly complex health care delivery problems and improve patient and population outcomes. Selected concepts to achieve these ends are introduced at a high level in this article. Readers are invited to explore concepts that resonate with their current situation in further detail.

Development of eConsult reflective learning tools for healthcare providers: a pragmatic mixed methods approach.

BACKGROUND: Electronic consultation (eConsult) programs are crucial components of modern healthcare that facilitate communication between primary care providers (PCPs) and specialists. eConsults between PCPs and specialists. They also provide a unique opportunity to use real-world patient scenarios for reflective learning as part of professional development. However, tools that guide and document learning from eConsults are limited. The purpose of this study was to develop and pilot two eConsult reflective learning tools (RLTs), one for PCPs and one for specialists, for those participating in eConsults. METHODS: We performed a four-phase pragmatic mixed methods study recruiting PCPs and specialists from two public health systems located in two countries: eConsult BASE in Canada and San Francisco Health Network eConsult in the United States. In phase 1, subject matter experts developed preliminary RLTs for PCPs and specialists. During phase 2, a Delphi survey among 20 PCPs and 16 specialists led to consensus on items for each RLT. In phase 3, we conducted cognitive interviews with three PCPs and five specialists as they applied the RLTs on previously completed consults. In phase 4, we piloted the RLTs with eConsult users. RESULTS: The RLTs were perceived to elicit critical reflection among participants regarding their knowledge and practice habits and could be used for quality improvement and continuing professional development. CONCLUSION: PCPs and specialists alike perceived that eConsult systems provided opportunities for self-directed learning wherein they were motivated to investigate topics further through the course of eConsult exchanges. We recommend the RLTs be subject to further evaluation through implementation studies at other sites.

Practice Adjustments Made by Family Physicians During the COVID-19 Pandemic.

INTRODUCTION: COVID-19 pandemic-related health care disruptions necessitated rapid adaptation among family physicians to safely meet patient needs while protecting themselves and their staff. On April 1, 2020, the American Board of Family Medicine (ABFM) introduced a COVID Performance Improvement (PI) activity for physicians to report on and receive Family Medicine certification credit for practice adjustments they made during the early stages of the pandemic. We aimed to understand the types of interventions implemented, and lessons physicians learned from the efforts. METHODS: We analyzed data from COVID-PI activities submitted by self-selected family physicians between April 1 and June 30, 2020. We summarized the COVID-related topics chosen for improvement and performed a qualitative content analysis on a random sample of open-text responses about lessons learned. RESULTS: The most common practice changes among 1259 unique COVID-PI activity submissions related to virtualization of patient visits, implementing new workflows, developing screening protocols, and obtaining and preserving personal protective equipment. We identified 12 themes regarding lessons learned, most commonly regarding patient and staff safety, modified practice processes and workflows, positive perceptions of and future plans for virtual visits, access to care, and patient satisfaction. Most submitters noted early successes with their interventions. CONCLUSION: A PI activity template designed for continuous board certification allowed family physicians to report on how they successfully implemented short term practice changes during the early stages of the COVID-19 pandemic. Reflections from this subset of physicians regarding lessons learned may prove useful in informing future COVID-19 related practice changes.

Training Future Family Physicians to Become Master Adaptive Learners.

This article examines the use of a concept that teaches learners how to learn in the context of family medicine residency training. We describe the four phases of this master adaptive learning framework and its place in educational theory and adaptive expertise, its implications for graduate medical education training and Accreditation Council for Graduate Medical Education competencies, as well as its role in imprinting family medicine residents for career-long learning. We lay out pragmatic strategies supporting this concept with a proposed curricular format for training in family medicine, including small group teaching methods, didactics, the clinic visit, faculty development and an optimal learning environment.

Implementing Competency-Based Medical Education in Family Medicine: A Narrative Review of Current Trends in Assessment.

BACKGROUND AND OBJECTIVES: The implementation of effective competency-based medical education (CBME) relies on building a coherent and integrated system of assessment across the continuum of training to practice. As such, the developmental progression of competencies must be assessed at all stages of the learning process, including continuing professional development (CPD). Yet, much of the recent discussion revolves mostly around residency programs. The purpose of this review is to synthesize the findings of studies spanning the last 2 decades that examined competency-based assessment methods used in family medicine residency and CPD, and to identify gaps in their current practices. METHODS: We adopted a modified form of narrative review and searched five online databases and the gray literature for articles published between 2000 and 2020. Data analysis involved mixed methods including quantitative frequency analysis and qualitative thematic analysis. RESULTS: Thirty-seven studies met inclusion criteria. Fourteen were formal evaluation studies that focused on the outcome and impact evaluation of assessment methods. Articles that focused on formative assessment were prevalent. The most common levels of educational outcomes were performance and competence. There were few studies on CBME assessment among practicing family physicians. Thematic analysis of the literature identified several challenges the family medicine educational community faces with CBME assessment. CONCLUSIONS: We recommend that those involved in health education systematically evaluate and publish their CBME activities, including assessment-related content and evaluations. The highlighted themes may offer insights into ways in which current CBME assessment practices might be improved to align with efforts to improve health care.

Rapid Retooling, Acquiring New Skills, and Competencies in the Pandemic Era: Implications and Expectations for Physician Continuing Professional Development.

The SARS-CoV-2 (COVID 19) pandemic has necessitated changes in health care delivery, including increases in delivery of care through asynchronous or virtual means, and deployment of clinicians in different teams and settings. Physical distancing and redeployment of clinicians has also necessitated changes in health care continuing professional development (CPD). Health care delivery and CPD is unlikely to fully return (in the near term, if at all) to pre-pandemic status. The authors raise questions and opportunities for development and provision of CPD during and after the pandemic.

Association Between Board Certification, Maintenance of Certification, and Surgical Complications in the United States.

Physician credentialing processes aim to improve patient safety and quality, but little research has examined their direct relationship with surgical outcomes. Using national Medicare claims for 2009 to 2013, the authors studied the association between board certification and completion of Maintenance of Certification (MOC) requirements and surgeon rates of complications for 8 elective procedures. Exemplar surgeons were defined as those in the lowest decile of complication rates, and outlier surgeons were those in the highest decile. The analysis included 1.9 million procedures performed by 14 598 surgeons (64% orthopedics, 17% general surgery, 11% urology, 7% neurosurgery). Board-certified surgeons were less likely to be outliers (odds ratio 0.79 [0.66-0.94]). However, completion of MOC was not associated with differences in complication rates in orthopedic surgery or urology. Incorporating additional assessment methods into MOC, such as video evaluation of technical skills, retraining on state-of-the-art care, and peer review, may facilitate further improvements in surgical quality.

Incorporation of Quality and Safety Principles in Maintenance of Certification: A Qualitative Analysis of American Board of Medical Specialties Member Boards.

Maintenance of Certification (MOC) is one way physicians demonstrate to the public that they are competent to deliver safe, high-quality medical care. The authors conducted a qualitative content analysis of information from 24 Member Boards of the American Board of Medical Specialties. A constant comparative approach was used to categorize themes of quality and safety and related concepts. All Boards (24/24) had public websites with formal documents and 23/24 had at least 1 representative respond to a survey. Sixty-three percent (15/24) of Boards had a patient safety requirement, and 96% (23/24) of Boards had a quality improvement requirement. Nearly all Boards incorporate quality improvement and most incorporate patient safety, but their assessment methods and level of inclusion on the topic vary. Sharing methods of incorporating quality and safety among certifying Boards will allow Member Boards to cater to the evolution of the MOC landscape and benefit from other Boards' experiences.

Can Maintenance of Certification Work? Associations of MOC and Improvements in Physicians' Knowledge and Practice.

PURPOSE: To summarize the findings of studies, conducted by individuals both internal and external to the American Board of Medical Specialties (ABMS) Member Boards, of the associations of Maintenance of Certification (MOC) and improvements in physicians' knowledge and patient care processes or outcomes. METHOD: The authors conducted a narrative review of studies identified by searching PubMed and Web of Science for English-language articles from the United States published between 2000 and May 2017. To be included, articles had to examine the relationship of MOC to physician knowledge, clinical practice processes, or patient care outcomes. The initial search yielded 811 articles. After two rounds of review and excluding those articles that did not fit the study criteria, 39 articles were included for analysis. RESULTS: The 39 included studies were conducted by or included diplomates of 12 ABMS Member Boards. Twenty-two studies examined MOC processes that were developed by an ABMS Board; 17 examined interventions that were developed by nonboard entities but accepted for MOC credit by an ABMS Board. Thirty-eight studies examined a single component of MOC; 24 studied the improvement in medical practice component. Thirty-seven studies reported at least one positive outcome. CONCLUSIONS: Most of the studies included in this review highlighted circumstances in which MOC was associated with positive impacts on physician knowledge and patient care processes or outcomes. Future collaborative research is needed to improve the relevance, helpfulness, and generalizability of continuing certification to different physicians across specialties and practice settings.

Longitudinal assessments in continuing specialty certification and lifelong learning.

Longitudinal assessment (LA) involves the regular, spaced delivery of a limited number of questions on practice relevant content on a computer or mobile internet platform. Depending on the platform, participants may indicate relevance of the content to their practice and confidence in their answer prior to receiving immediate feedback (including critiques) on each question. Individual dashboards may be included to assist participants in tracking progress and identifying areas of strength and weaknesss across a content blueprint. This paper provides an overview of the theoretical underpinnings underlying LA programs, briefly describes current uses of LA in medicine and suggests areas for evaluating the role of LA in continuing medical specialty certification and continuing professional development.

Depressive Symptoms, Antidepressant Medication Use, and Inflammatory Markers in the Diabetes Prevention Program.

OBJECTIVE: Antidepressant medication use (ADM) has been shown to predict diabetes. This article assessed the role of inflammatory markers in this relationship within the Diabetes Prevention Program (DPP). METHODS: DPP participants randomized to metformin (MET), life-style intervention (ILS), or placebo (PLB) were assessed for depression (Beck Depression Inventory [BDI]) annually, ADM use semiannually, serum inflammatory markers (C-reactive protein [CRP], interleukin 6 [IL-6]) at baseline and year 1, and diagnosis of type 2 diabetes mellitus (T2DM) semiannually (for 3.2 years). RESULTS: At baseline (N = 3187), M (SD) body mass index was 34 (6) kg/m and the median (interquartile range) BDI score was 3 (1-7). One hundred eighty-one (5.7%) reported ADM use and 328 (10%) had BDI scores of 11 or higher. CRP and IL-6 levels did not differ by treatment group. Baseline ADM, but not BDI score, was associated with higher levels of baseline CRP adjusted for demographic, anthropometric variables, and other medications (20% higher, p = .01). Year 1 CRP decreased for non-ADM users in the MET (-13.2%) and ILS (-34%) groups and ADM users in the ILS group (-29%). No associations were found with IL-6. CRP and continuous use of ADM predicted incident T2DM in the PLB group. In the ILS group, continuous and intermittent ADM, but not CRP, predicted T2DM. In the MET group, CRP predicted incident T2DM. CRP did not mediate the risk of T2DM with ADM use in any group. CONCLUSIONS: ADM was significantly associated with elevated CRP and incident T2DM. In the PLB group, ADM and CRP independently predicted onset of T2DM; however, CRP did not significantly mediate the effect of ADM.

Educational Technologies for Physician Continuous Professional Development: A National Survey.

PURPOSE: To determine the past experiences with, current use of, and anticipated use of online learning and simulation-based education among practicing U.S. physicians, and how findings vary by age. METHOD: The authors surveyed 4,648 randomly sampled board-certified U.S. physicians, September 2015 to April 2016, using Internet-based and paper questionnaires. Survey items (some optional) addressed past and current technology usage, perceived technology effectiveness, and anticipated future use of specific technology innovations. RESULTS: Of 988 respondents, 444 completed optional items. Of these, 429/442 (97.1%) had used online learning and 372/442 (84.2%) had used simulation-based education in the past five years. Desire for more online learning was modest (mean [standard deviation], 4.6 [1.5]; 1 = strongly disagree, 7 = strongly agree), as was desire for more simulation-based education (4.2 [1.7]). Both online learning and simulation-based education were perceived as effective (5.2 [1.4]; 5.0 [1.4]). Physicians believed they possess adequate skills for online learning (5.8 [1.2]) and that point-of-care learning is vital to effective patient care (5.3 [1.3]). Only 39.0% used objective performance data to guide their learning choices, although 64.6% agreed that such information would be useful. The highest-rated innovations included a central repository for listing educational opportunities and tracking continuing education credits, an app to award credit for answering patient-focused questions, 5-minute and 20-minute clinical updates, and an e-mailed "question of the week." Responses to most survey items were similar across age groups. CONCLUSIONS: Practicing physicians generally seem receptive and prepared to use a variety of educational technologies, regardless of age.

Factors Influencing Physicians' Selection of Continuous Professional Development Activities: A Cross-Specialty National Survey.

INTRODUCTION: We sought to understand what influences physicians' decisions about participation in continuous professional development (CPD) activities, and how often physicians engage in specific CPD activities. METHODS: From September 2015 to April 2016, we administered a survey to 4648 randomly sampled licensed US physicians. Survey items addressed perceived barriers to CPD, factors that might influence participation in four prototypical CPD activities (reading an article, or completing a local activity, online course, or far-away course), and frequency of CPD engagement. RESULTS: Nine hundred eighty-eight (21.6%) physicians responded. The most important barriers were time (mean [SD] 3.5 [1.3], 1 = not important, 5 = extremely important) and cost (2.9 [1.3]). In prioritizing factors influencing participation in four prototypical CPD activities, topical relevance consistently had the highest average rank. Quality of content and time to complete the activity were also frequently selected. Over the past 3 years, most physicians reported having participated in patient-focused learning and self-directed learning on a weekly basis; quality improvement and local continuing medical education (CME) activities several times per year; online learning, on-site courses, and national board-related activities a few times per year; and interprofessional learning less than once per year. Physicians believed that they ought to engage more often in all of these activities except board-related activities. They would like CME credit for these activities much more often than currently obtained. DISCUSSION: The reasons physicians select a given CPD activity vary by activity, but invariably include topic and quality of content. Physicians want CME credit for the CPD activities they are already doing.