Building a core competency assessment program for all stakeholders: the design and building of sailing ships can inform core competency frameworks.

When educators are developing an effective and workable assessment program in graduate medical education by employing action research and stakeholder mapping to identify core competency domains and directives, the multi-stage process can be guided and informed by utilizing the story of designing, building and sea-testing sailing ships as a metaphor. However, the current challenge of physician burnout demands additional attention when formulating medical training frameworks, assessment guidelines and mentoring programs in 2020. The possibility of job-crafting is raised for consideration by designers of core competency frameworks in the health professions.

Specialty Board Certification Rate as an Outcome Metric for GME Training Institutions: A Relationship With Quality of Care.

Educational outcome measures, known to be associated with the quality of care, are needed to support improvements in graduate medical education (GME). This retrospective observational study sought to determine whether there was a relationship between the specialty board certification rates of GME training institutions and the quality of care delivered by their graduates. It is based on 7 years of hospitalizations in Pennsylvania (N = 354,767) with diagnoses of acute myocardial infarction, congestive heart failure, gastrointestinal hemorrhage, or pneumonia. The 2,265 attending physicians were self-identified internists, and they completed their training in 59 institutions. The percentage of board-certified physicians from each training institution, excluding the physician herself or himself, was calculated and an indicator of whether it exceeded 80% was created. This was analyzed against inhospital mortality and length of stay, adjusted for patient/physician/hospital characteristics. There were significantly lower odds of mortality (adjusted Odd's ratio [OR] = .92, 95% CI [0.86, 0.98]) and log length of stay (adjusted OR = .98, 95% CI [.94, .99]) when the attending physician trained in a residency program with an 80% or greater certification rate. The results suggest that specialty certification rates may be a useful educational outcome for residency training programs.

A Further Examination of Previous and Future Policy Opportunities of the Educational Commission for Foreign Medical Graduates.

The Educational Commission for Foreign Medical Graduates (ECFMG) has a distinguished history of providing high-quality, innovative products and services to international medical graduates (IMGs) seeking to study and practice medicine in the United States. In 2010, the ECFMG board introduced a policy stating that, starting in 2023, all IMGs applying to the ECFMG for credentialing must have graduated from a medical school that has been accredited by an internationally recognized accrediting body akin to the Liaison Committee on Medical Education in the United States or the World Federation for Medical Education. In this issue of Academic Medicine, Tackett reviews the reasons for the policy and its adoption worldwide. After eight years, the number of schools meeting the new standard is modest. He is concerned about the negative effect a continuing low rate of adoption will have on U.S. postgraduate medical education programs and workforce supply. The author of this Invited Commentary offers three perspectives: an overview of the ECFMG's successes, alternative measurement tools to ensure the quality of IMGs entering the United States, and frameworks by which an organization like the ECFMG can refine its policy positions and processes for the future. Academia can expect the ECFMG, given its history of successful collaboration and public accountability, to continue using best practices and to adjust policies according to evidence. As a publicly accountable authority, the ECFMG should debrief key stakeholders on current policies, track IMG practice patterns, and share the resulting data with stakeholders to inform their IMG-related planning decisions.

Considerations that will determine if competency-based assessment is a sustainable innovation.

Educational assessment for the health professions has seen a major attempt to introduce competency based frameworks. As high level policy developments, the changes were intended to improve outcomes by supporting learning and skills development. However, we argue that previous experiences with major innovations in assessment offer an important road map for developing and refining assessment innovations, including careful piloting and analyses of their measurement qualities and impacts. Based on the literature, numerous assessment workshops, personal interactions with potential users, and our 40 years of experience in implementing assessment change, we lament the lack of a coordinated approach to clarify and improve measurement qualities and functionality of competency based assessment (CBA). To address this worrisome situation, we offer two roadmaps to guide CBA's further development. Initially, reframe and address CBA as a measurement development opportunity. Secondly, using a roadmap adapted from the management literature on sustainable innovation, the medical assessment community needs to initiate an integrated plan to implement CBA as a sustainable innovation within existing educational programs and self-regulatory enterprises. Further examples of down-stream opportunities to refocus CBA at the implementation level within faculties and within the regulatory framework of the profession are offered. In closing, we challenge the broader assessment community in medicine to step forward and own the challenge and opportunities to reframe CBA as an innovation to improve the quality of the clinical educational experience. The goal is to optimize assessment in health education and ultimately improve the public's health.

Patients of doctors further from medical school graduation have poorer outcomes.

CONTEXT: There is an apparent contradiction between the findings of studies indicating that patient outcomes are better when physicians have a greater volume of practice and those that find outcomes to be worse with increased time since training, which implies greater volume. OBJECTIVES: This study was designed to estimate the adjusted relationships between physicians' characteristics, including recent practice volume and time since medical school graduation, and patient outcomes. METHODS: This is a retrospective observational study based on all Pennsylvania hospitalisations over 7 years for acute myocardial infarction, congestive heart failure, gastrointestinal haemorrhage, hip fracture and pneumonia. It refers to 694 020 hospitalisations in 184 hospitals attended by 5280 internists and family physicians. Patient severity of illness at admission and in-hospital mortality, hospital location and volume, and the physician's recent practice volume, time since medical school graduation, board certification, and citizenship or medical school location were analysed. RESULTS: After adjustment, recent practice volume did not have a statistically significant association with in-hospital mortality for all of the conditions combined. By contrast, each decade since graduation from medical school was associated with a 4.5% increase in relative risk for patient mortality. CONCLUSIONS: Recent practice volume does not mitigate the increase in patient mortality associated with physicians' time since medical school graduation. These findings underscore the need to finds ways to support and encourage learning.

The role of theory-based outcome frameworks in program evaluation: Considering the case of contribution analysis.

In an era demanding greater accountability and the demonstration of positive outcomes and impacts, the field for the evaluation of interventions, program development and outreach projects is being challenged in many fields, including education, medical care, public health and social development. In consequence, the leaders in this field significantly changed their approaches to the evaluation of such interventions. Evaluators noted that simple linear models of evaluation do not address the wider community of interests and stakeholders involved in today's innovative and wide-reaching programs. Moreau raises the possible usefulness of contribution analysis in responding to the calls for broader accountability. In this commentary, the elements of these emerging approaches are reviewed and explained for teachers with reporting responsibilities in health sciences education. The presentation is intended to expand on Moreau's argument and suggestions such that educators may be able to consider the use of theory-based evaluations, such as contribution analyses, in the evaluation of their institutional programs and interventions. These possible applications are especially relevant to the increasingly more complicated and complex interventions that characterize many of the educational interventions as more health profession programs are moved into and impact on the larger societal community.

The relationship between licensing examination performance and the outcomes of care by international medical school graduates.

PURPOSE: The Step 2 Clinical Knowledge (CK) examination of the United States Medical Licensing Examination sequence is a requirement for the certification of international medical graduates (IMGs) by the Educational Commission for Foreign Medical Graduates. An association between scores on the test and the quality of care later provided by those who take it is central to its use in certification and licensure. The purpose of this study was to determine whether there is a relationship between scores on Step 2 CK and patient outcomes for IMGs. METHOD: This is a retrospective observational study of the 60,958 hospitalizations from 2003 to 2009 in Pennsylvania where the principal diagnosis was acute myocardial infarction or congestive heart failure and the attending physician (N = 2,525) was an IMG who had taken the Step 2 CK. The main measures were the three-digit scores on Step 2 CK and in-hospital mortality. RESULTS: After adjustment for severity of illness, physician characteristics, and hospital characteristics, performance on Step 2 CK had a statistically significant inverse relationship with mortality. Each additional point on the examination was associated with a 0.2% (95% CI: 0.1%-0.4%) decrease in mortality. The size of the effect is noteworthy, with each standard deviation (roughly 20 points) equivalent to a 4% change in mortality risk. CONCLUSIONS: These findings provide evidence for the validity of Step 2 CK scores. Given the magnitude of its relationship with patient outcomes, the results support the use of the examination as an effective screening strategy for licensure.

Outcomes of cardiac surgery: associations with physician characteristics, institutional characteristics, and transfers of care.

BACKGROUND: Although there are several studies of the human and system factors that influence the outcomes of cardiac surgery, it remains difficult to draw conclusions because many do not simultaneously adjust for the characteristics of patients, physicians, and institutions. The current study explores the associations between these factors and inhospital mortality, with a particular focus on whether patients had the same operating and attending physician. METHOD AND RESULTS: This is a retrospective observational study of 114,751 hospitalizations from 2003 to 2009 in Pennsylvania that included a coronary artery bypass graft, valve surgery, or both. The study included 70 teaching and nonteaching hospitals, 289 operating physicians who were also the attending physicians for 75% of the hospitalizations, and 2654 attending physicians for the remaining hospitalizations. After adjustment, there was a 38.4% decrease (95% CI, 20.3%-56.5%) in mortality when the operating and attending physician were the same. For the operator, each procedure performed was associated with a 0.05% (95% CI, 0.04%-0.06%) decrease in mortality and each year since medical school was associated with a 0.9% (95% CI, 0.02%-1.8%) increase in mortality. For the attending, each year since medical school was associated with a 0.67% (95% CI, 0.01%-1.4%) decrease in patient mortality. CONCLUSIONS: The findings indicated that an increase in the log odds of mortality was associated with the transfer of care between an attending and operating physician. Better patient outcomes were associated with an operator with higher volume who was closer to medical school graduation and an attending physician with more clinical experience.

Educators must consider patient outcomes when assessing the impact of clinical training.

CONTEXT: The concept of outcomes has been used in health care for over 140 years. The use of outcomes in assessing quality of care regained prominence in the 1960s based on Donabedian's framework of structures, processes and outcomes. In the 1990s, the use of outcomes in medical education gained great favour, although the outcomes used were not carefully defined. Recently, a debate has ensued about the costs and, thus, sustainability of current health care programmes, focusing on the (non-)necessity of services, missed prevention opportunities and the efficiency of treatment programmes. Measurements using education outcomes and health care outcomes must take these issues into account, preferably from a common framework. As health care becomes increasingly costly and even inefficient, issues of effectiveness are often neglected in policy making. METHODS: This paper uses peer-reviewed evidence and an outcomes framework to explore the implications of current realities for the makers of education policy in the health professions and for the staff who train health professionals. DISCUSSION: If the ultimate impacts of practices and policies in health professions education are not considered, how will we know if our education structures, processes and outcomes are optimal? This essay examines this question from the perspectives of three related issues. The first refers to the need for a common framework if the outcomes of patient and community care are to be evaluated properly. The second perspective refers to whether it is feasible to consider both patient-based outcomes and patient-reported outcomes in assessing the impact of education programmes, especially at more advanced levels of training. The third perspective concerns the challenges and limitations that may be encountered in focusing on patient outcomes as a measure of the impact of education. The concluding discussion suggests how the results of such longer-term impact studies should be interpreted as key validity checks on the quality and effectiveness of medical education and clinical education if we are to address the validity and efficiency of outcomes used in education and training.

A framework for designing, implementing, and sustaining a national simulation network: building incentive-based network structures and iterative processes for long-term success: the case of the Medical Council of Canada's Qualifying Examination, Part II.

The use of networks for sharing and distributing information, for institutional collaboration, and action programs is commonplace. In 1989, the Medical Council of Canada began the implementation of a national clinical licensing examination to assess physicians for practice skills and decision making using standardized or simulated patients in an Objective Structured Clinical Examination format. Once fully implemented, the examination was administered through a network of medical schools at 16 locations across Canada in two languages twice yearly. That network has functioned successfully for 17 years. This article reviews the literature and examines the reasons and incentives for the long-term sustainability of the network. Based on that assessment, a framework is presented for analyzing, designing, and sustaining a national simulation network. It emphasizes the need for an iterative approach and identifies the success factors that can facilitate the adoption of a national simulation network for use in professional credentialing and licensure.

Evidence-based practice: a review of theoretical assumptions and effectiveness of teaching and assessment interventions in health professions.

Health care professionals are expected to use a systematic approach based on evidence, professional reasoning and client preferences in order to improve client outcomes. In other words, they are expected to work within an evidence-based practice (EBP) context. This expectation has had an impact on occupational therapy academic programs' mandates to prepare entry-level clinicians who demonstrate competence in the knowledge, skills and behaviors for the practice of evidence-based occupational therapy. If the EBP approach is to be entrenched in the day to day practice of future clinicians, a pedagogically sound approach would be to incorporate EBP in every aspect of the curriculum. This, however, would require a comprehensive understanding of EBP: its basis, the principles that underpin it and its effectiveness in promoting core professional competencies. The existing literature does not elucidate these details nor does it shed light on how requisite competencies for EBP are acquired in professional education in general and in occupational therapy education in particular. Drawing from educational psychology and EBP in the health professions, this paper provides a critical review of the evidence that supports EBP and the effectiveness of EBP teaching and assessment interventions in professional heath sciences programs and offers suggestions for the design of EBP instruction, grounding recommendations in educational theory for the health professions.

Evaluating the quality of care provided by graduates of international medical schools.

One-quarter of practicing physicians in the United States are graduates of international medical schools. The quality of care provided by doctors educated abroad has been the subject of ongoing concern. Our analysis of 244,153 hospitalizations in Pennsylvania found that patients of doctors who graduated from international medical schools and were not U.S. citizens at the time they entered medical school had significantly lower mortality rates than patients cared for by doctors who graduated from U.S. medical schools or who were U.S. citizens and received their degrees abroad. The patient population consisted of those with congestive heart failure or acute myocardial infarction. We found no significant mortality difference when comparing all international medical graduates with all U.S. medical school graduates.

Commentary: one strategy for building public trust in changing times.

Major health care reforms are being debated in the United States. While these debates address issues of access and cost, the systems-based problems of patient safety, continuous quality improvement, and an integrated approach to continuing professional development (CPD) remain traditional opportunities for the profession to directly improve health care and maintain professional accountability. Such challenges can be addressed independently of proposed reforms and offer an opportunity for the profession to build greater public trust. Given recent evidence questioning many assumptions behind individually focused CPD, and as physicians' work shifts into group and team contexts, it is an opportune time to address better CPD strategies within emerging group and team settings.The first strategic change requires a focus on managing the development of the individual physician's educational growth into a systems-oriented approach based on better information and feedback within groups of practitioners and health care teams. Second, the renewal of the linked visions of professional collegiality and accountability with professional regulation needs to be seen as a collective responsibility across key organizations within the profession's normal accountability framework. Thus, the professional colleges, certifying boards, and regulatory authorities need to collaborate with the CDP community in refocusing their collective activities to promote the profession's traditional role of improving the quality of care and maintaining the public's trust in these times of intense policy debate.

The circle game: understanding physician migration patterns within Canada.

This report explores the movement of physicians to, from, and within Canada and identifies recurring patterns of migration. The primary position of the report is that physician movement is part of reality both internationally and within Canada, and that movement of Canadian-trained physicians creates a need for international medical graduates (IMGs) in "physician-losing" locations. The report's argument is based on data retrieved from public sources on aggregate physician practice patterns in Canada and analyzed for migration patterns. In addition, literature was reviewed on factors affecting the migration patterns being described.Canadian-educated physicians have tended to move from less prosperous to more prosperous provinces and from rural to urban areas; because of the resulting need, the physician-losing locales generally have the highest proportions of IMGs. Physicians traditionally have tended to emigrate from Canada to the United States, thus increasing Canadian demand for IMGs, but recently this movement has slowed and even reversed. In Canada, liberalized immigration policies for physicians combined with a shortage of postgraduate training positions to create a serious bottleneck early in the current decade. However, this problem is now being resolved. In summary, physician migration within Canada shows specific long-term patterns, and IMGs will be needed in underserved areas for years to come. Well-informed policies for workforce management are essential in Canada to ensure an adequate physician supply consisting mainly of Canadian-educated physicians but also including IMGs. A role for nonadvocacy groups such as the Educational Commission for Foreign Medical Graduates may be to help ensure that recruitment of physicians from developing countries follows accepted ethical principles.

Medical workforce policy-making in Canada, 1993-2003: reconnecting the disconnected.

The authors set out to review Canadian medical workforce policies for 1993 to 2003 and assess if data existed in the 1990s that could have reversed the policy decision to curtail the supply of physicians from Canada's medical schools just as Canada was about to experience a developing shortage. The authors reviewed existing descriptive data sources regarding Canadian physician workforce size and activity from 1986 to 2003, including the Canadian Medical Association workforce database. The review indicated that a significant loss of physicians to retirement was imminent. Physician workforce productivity had started to fall by the early 1990s. Emigration to the United States had risen above traditional levels in the early 1990s and remained higher into the late 1990s. Despite these existing findings, an integrated adjustment to physician workforce policies taken in 1993-94 only occurred after 1999. The authors recommend that policy makers and managers must monitor the numbers from existing sources. To optimize these sources, planned data tracking and linkages are essential. The period in question demonstrated major disconnects in coordinating implementation, wherein subject experts monitoring data trends were not adequately utilized by policy makers. Finally, in complex systems with regional differences, policy decisions based on normative data are insufficient.