Physician Champions of Quality and Safety: Perspectives of MD Quality and Safety Leads to Drive Quality Innovation at an Academic Hospital.

While the importance of physician involvement in organizational quality and safety (Q&S) activities has been well established, a paucity of information exists on tangible supports needed to effectively execute this role. Interviews with 13 MD Q&S leads uncovered common enablers, including valuing Q&S work academically, hiring skilled collaborators, ensuring appropriate power and authority to advance Q&S initiatives, facilitating connections, emphasizing culture change and strong action by leadership. To operationalize these enablers and drive quality innovation, organizations should prioritize the identification and appointment of MD Q&S leads for each department/division and facilitate their assembly as a formal physician Q&S committee.

An Education Framework for Effective Implementation of a Health Information System: Scoping Review.

BACKGROUND: To optimize their use of a new Health Information System (HIS), supporting health care providers require effective HIS education. Failure to provide this education can significantly hinder an organization's HIS implementation and sustainability efforts. OBJECTIVE: The aim of this review is to understand the most effective educational strategies and approaches to enable health care providers to optimally use an HIS. METHODS: Ovid MEDLINE, Ovid Embase, EBSCO Cumulative Index to Nursing and Allied Health Literature, and EBSCO Education Resources Information Center were searched to identify relevant papers. Relevant studies were systematically reviewed and analyzed using a qualitative thematic analysis approach. RESULTS: Of the 3539 studies screened, 17 were included for data extraction. The literature on the most effective approaches to enable health care providers to optimally use an HIS emphasized the importance of investing in engaging and understanding learners in the clinical context, maximizing the transfer of learning to care, and designing continuous and agile evaluation to meet the emerging demands of the clinical environment. CONCLUSIONS: This review supports the advancement of a new HIS learning framework that organizational leaders and educators can use to guide HIS education design and development. Future research should examine how this framework can be translated into practice.

Beyond the cultural myth of medical meritocracy.

BACKGROUND: We examine the cultural myth of the medical meritocracy, whereby the "best and the brightest" are admitted and promoted within the profession. We explore how this narrative guides medical practice in ways that may no longer be adequate in the contexts of practice today. METHODS: Narrative analysis of medical students' and physicians' stories. RESULTS: Hierarchies of privilege within medicine are linked to meritocracy and the trope of the "hero's story" in literature. Gender and other forms of difference are generally excluded from narratives of excellence, which suggests operative mechanisms that may be contributory to observed differences in attainment. We discuss how the notion of diversity is formulated in medicine as a "problem" to be accommodated within merit, and posit that medical practice today requires a reformulation of the notion of merit in medicine, valorising a diversity of life experience and skills, rather than "retrofitting" diversity concerns as problems to be accommodated within current constructs of merit. CONCLUSIONS: Three main action-oriented outcomes for a better formulation of merit relevant to medical practice today are suggested: (a) development of assessors' critical consciousness regarding the structural issues in merit assignment; (b) alignment of merit criteria with relevant societal outcomes, and (c) developing inclusive leadership to accommodate the greater diversity of excellence needed in today's context of medical practice. A reformulation of the stories through which medical practitioners and educators communicate and validate aspects of medical practice will be required in order for the profession to continue to have relevance to the diverse societies it serves.

The Karolinska Institutet Prize for Research in Medical Education: A history.

Purpose: This article presents a history of the Karolinska Institutet Prize for Medical Education (KIPRIME), highlighting the history of, and influences on, its funders Drs. Gunnar Höglund and Anna-Stina Malmborg.Methods: Historic analysis of an archive of documents developed by the authors in a prior study exploring philanthropy in medical education research. Documents in the archive were drawn from publicly available Internet sources, media reports about the KIPRIME and its winners and an interview with Drs. Höglund's and Malmborg. The latter interview was conducted with Ethics Board approval in non-anonymous fashion and with the explicit permission of the interviewees to present their personal information and to cite their words. Finally, observations were shaped iteratively by the authors on multiple trips to the Karolinska Institutet with input from the KIPRIME prize committee leaders.Findings: The results of this analysis present a history of the prize situating it in the personal histories of, and influences acting upon, Drs. Höglund and Malmborg. Special attention is given to the potential influence of the Nobel Prizes and the culture of philanthropy in Sweden.

Assessment of professionalism: From where have we come - to where are we going? An update from the Ottawa Consensus Group on the assessment of professionalism.

BACKGROUND: In 2009, an International Working Group (IWG) on the Assessment of Professionalism began collaborating and published recommendations in 2011. Nearly a decade later the IWG reconvened to take stock of the state of practice and research in professionalism and the impact of the 2011 report. METHOD: A bibliometric study identified all publications on assessment of professionalism since 2011, noting those that cited the original report. Articles were coded to identify the reason for citation and new trends in assessment. Bibliometric data were supplemented by discussion groups held at key international education meetings. RESULTS: Six-hundred publications on the assessment of professionalism were found in Google Scholar and 164 in Web of Science since 2011, of which 177 (30%) and 84 (50%) respectively cited the original IWG publication. English language publications were most common (83%), but there were articles in 13 other languages by authors from 40 countries. The report was cited commonly to justify attention to professionalism in general (41%), assessment of professionalism (38%) and to explore professionalism in different countries and professions (25%). A thematic analysis showed that of 9 research areas recommended in 2011, 7 of 9 categories were represented with a large increase in research across languages and cultures. CONCLUSIONS: Though the assessment of professionalism remains a challenge the research base continues to grow, especially related to professionalism across cultures and languages, and a large percentage of publications cite the IWG recommendations. There remains a gap in research and writing about patients' perspectives.

Philanthropy in health professions education research: determinants of success.

CONTEXT: Fund-raising is a new practice in medical education research. OBJECTIVES: This qualitative study explores a cross-sectional analysis of philanthropy in medical education in Canada and Europe and identifies some common characteristics in the fund-raising system, key roles and characteristics of research sites that have had success. METHODS: Medical education research sites that had received donations greater than Can$100 000 were identified by searching publicly available sources. Interviews were conducted with 25 individuals from these and other sites, in four categories: medical education leaders (n = 9); philanthropy-supported chairholders and researchers (n = 5); donors of over Can$100 000 (n = 7), and advancement professionals (n = 4). Interview transcripts were inductively coded to identify themes. RESULTS: Five factors associated with success in accessing philanthropic sources were identified in the sample: support of the organisation's senior leadership; a charismatic champion who motivates donors; access to an advancement office or foundation; impetus to find funds beyond traditional operating budgets, and understanding of the conceptual and practical dimensions of fund-raising. Three types of donor (medical education insider, donor collective and general philanthropist), four faculty roles (trailblazers, rock stars, 'Who? Me?' people and future fund-raisers) and six stages in the fund-raising cycle were also identified. CONCLUSIONS: Philanthropy is a source of funding with the potential to significantly advance education research. Yet competence in fund-raising is not widely developed among medical education research leaders. Successful accessing of philanthropic sources of funding requires the ability to articulate the impact of philanthropy in medical education research in a way that will interest donors. This appears to be challenging for medical education leaders, who tend to frame their work in academic terms and have trouble competing against other fund-raising domains. Medical education research institutes and centres will benefit from developing greater understanding of the conception and practices of fund-raising.

Validity: one word with a plurality of meanings.

Validity is one of the most debated constructs in our field; debates abound about what is legitimate and what is not, and the word continues to be used in ways that are explicitly disavowed by current practice guidelines. The resultant tensions have not been well characterized, yet their existence suggests that different uses may maintain some value for the user that needs to be better understood. We conducted an empirical form of Discourse Analysis to document the multiple ways in which validity is described, understood, and used in the health professions education field. We created and analyzed an archive of texts identified from multiple sources, including formal databases such as PubMED, ERIC and PsycINFO as well as the authors' personal assessment libraries. An iterative analytic process was used to identify, discuss, and characterize emerging discourses about validity. Three discourses of validity were identified. Validity as a test characteristic is underpinned by the notion that validity is an intrinsic property of a tool and could, therefore, be seen as content and context independent. Validity as an argument-based evidentiary-chain emphasizes the importance of supporting the interpretation of assessment results with ongoing analysis such that validity does not belong to the tool/instrument itself. The emphasis is on process-based validation (emphasizing the journey instead of the goal). Validity as a social imperative foregrounds the consequences of assessment at the individual and societal levels, be they positive or negative. The existence of different discourses may explain-in part-results observed in recent systematic reviews that highlighted discrepancies and tensions between recommendations for practice and the validation practices that are actually adopted and reported. Some of these practices, despite contravening accepted validation 'guidelines', may nevertheless respond to different and somewhat unarticulated needs within health professional education.

Twelve tips for undertaking reflexive global health experiences in medicine.

BACKGROUND: While interest and opportunities for global health experiences (GHE) continue to grow, the preparation of students and health professionals alike to engage in these GHEs remains limited. AIMS: This article provides tips for reflexivity prior to undertaking a GHE and suggests ways to debrief the experience in order to ensure that trainees and professionals that engage in GHEs can both help their intended communities and also get the most out of the experience. METHODS: The authors conducted a scoping review using Medline, PubMed and Google scholar using searching the terms: global health, global health experience, global health research, and international medical elective. We supplemented this search with our own experiences working with international partners. CONCLUSIONS: GHEs should be undertaken with reflexivity prior to, during and subsequent to the experience in order to ensure that all collaborators in the partnership meet their intended goals.

Toward a research agenda for competency-based medical education.

Competency-based medical education (CBME) is both an educational philosophy and an approach to educational design. CBME has already had a broad impact on medical schools, residency programs, and continuing professional development in health professions around the world. As the CBME movement evolves and CBME programs are implemented, a wide range of emerging research questions will warrant scholarly examination. In this paper, we describe a proposed CBME research agenda developed by the International CBME Collaborators. The resulting framework includes questions about the meaning of key concepts of CBME and their implications for learners, faculty members, and institutional structures. Other research questions relate to the learning process, the meaning of entrustment decisions, fundamental measurement issues, and the nature and definition of standards. The exploration of these questions will help to solidify the theoretical foundation of CBME, but many issues related to implementation also need to be addressed. These pertain to, among other things, nurturing independent learning, assembling and using assessment results to make decisions about competence, structuring feedback, supporting remediation, and how best to evaluate the longer-term outcomes of CBME. High-quality research on these questions will require rigorous outcome measures with strong validity evidence. The complexity of CBME necessitates theoretical and methodological diversity. It also requires multi-institutional studies that examine effects at multiple levels, from the learner to the team, the institution, and the health care system. Such a framework of research questions can guide and facilitate scholarly discourse on the theoretical and practical body of knowledge related to competency-based health professions education.

The show must go on? Patients, props and pedagogy in the theatre of the OSCE.

According to Shakespeare, all the world's a stage, and all the men and women merely players. The objective structured clinical examination (OSCE), that most ubiquitous form of assessment in health professions education, offers us a particular instance of this maxim. Comprising at first glance a world of psychometric data, detailed checklists and global rating scales, the OSCE sets out to facilitate the assessment of a candidate's competence in a highly standardised and objective fashion. Despite this clear intention, OSCEs also offer a rich vein of (often unacknowledged) social and cultural processes. In this commentary, we draw on Goffman's dramaturgy metaphor and our experiences to undertake a wry examination of some of the least intended consequences of OSCEs. We take a satirical look at both the potential impact on patients and the pedagogical implications of this form of assessment. We now urge you to sit back, settle in and enjoy the show, as we raise the curtain on this one-night-only performance!

Use of an Objective Structured Assessment of Technical Skill After a Sports Medicine Rotation.

PURPOSE: The purpose of this study was to determine if the use of an Objective Structured Assessment of Technical skill (OSATS), using dry models, would be a valid method of assessing residents' ability to perform sports medicine procedures after training in a competency-based model. METHODS: Over 18 months, 27 residents (19 junior [postgraduate year (PGY) 1-3] and 8 senior [PGY 4-5]) sat the OSATS after their rotation, in addition to 14 sports medicine staff and fellows. Each resident was provided a list of 10 procedures in which they were expected to show competence. At the end of the rotation, each resident undertook an OSATS composed of 6 stations sampled from the 10 procedures using dry models-faculty used the Arthroscopic Surgical Skill Evaluation Tool (ASSET), task-specific checklists, as well as an overall 5-point global rating scale (GRS) to score each resident. Each procedure was videotaped for blinded review. RESULTS: The overall reliability of the OSATS (0.9) and the inter-rater reliability (0.9) were both high. A significant difference by year in training was seen for the overall GRS, the total ASSET score, and the total checklist score, as well as for each technical procedure (P < .001). Further analysis revealed a significant difference in the total ASSET score between junior (mean 18.4, 95% confidence interval [CI] 16.8 to 19.9) and senior residents (24.2, 95% CI 22.7 to 25.6), senior residents and fellows (30.1, 95% CI 28.2 to 31.9), as well as between fellows and faculty (37, 95% CI 36.1 to 27.8) (P < .05). CONCLUSIONS: The results of this study show that an OSATS using dry models shows evidence of validity when used to assess performance of technical procedures after a sports medicine rotation. However, junior residents were not able to perform as well as senior residents, suggesting that overall surgical experience is as important as intensive teaching. CLINICAL RELEVANCE: As postgraduate medical training shifts to a competency-based model, methods of assessing performance of technical procedures become necessary.

Validity of a new assessment rubric for a short-answer test of clinical reasoning.

BACKGROUND: The validity of high-stakes decisions derived from assessment results is of primary concern to candidates and certifying institutions in the health professions. In the field of orthopaedic manual physical therapy (OMPT), there is a dearth of documented validity evidence to support the certification process particularly for short-answer tests. To address this need, we examined the internal structure of the Case History Assessment Tool (CHAT); this is a new assessment rubric developed to appraise written responses to a short-answer test of clinical reasoning in post-graduate OMPT certification in Canada. METHODS: Fourteen physical therapy students (novices) and 16 physical therapists (PT) with minimal and substantial OMPT training respectively completed a mock examination. Four pairs of examiners (n = 8) participated in appraising written responses using the CHAT. We conducted separate generalizability studies (G studies) for all participants and also by level of OMPT training. Internal consistency was calculated for test questions with more than 2 assessment items. Decision studies were also conducted to determine optimal application of the CHAT for OMPT certification. RESULTS: The overall reliability of CHAT scores was found to be moderate; however, reliability estimates for the novice group suggest that the scale was incapable of accommodating for scores of novices. Internal consistency estimates indicate item redundancies for several test questions which will require further investigation. CONCLUSION: Future validity studies should consider discriminating the clinical reasoning competence of OMPT trainees strictly at the post-graduate level. Although rater variance was low, the large variance attributed to error sources not incorporated in our G studies warrant further investigations into other threats to validity. Future examination of examiner stringency is also warranted.

How to set the bar in competency-based medical education: standard setting after an Objective Structured Clinical Examination (OSCE).

BACKGROUND: The goal of the Objective Structured Clinical Examination (OSCE) in Competency-based Medical Education (CBME) is to establish a minimal level of competence. The purpose of this study was to 1) to determine the credibility and acceptability of the modified Angoff method of standard setting in the setting of CBME, using the Borderline Group (BG) method and the Borderline Regression (BLR) method as a reference standard; 2) to determine if it is feasible to set different standards for junior and senior residents, and 3) to determine the desired characteristics of the judges applying the modified Angoff method. METHODS: The results of a previous OSCE study (21 junior residents, 18 senior residents, and six fellows) were used. Three groups of judges performed the modified Angoff method for both junior and senior residents: 1) sports medicine surgeons, 2) non-sports medicine orthopedic surgeons, and 3) sports fellows. Judges defined a borderline resident as a resident performing at a level between competent and a novice at each station. For each checklist item, the judges answered yes or no for "will the borderline/advanced beginner examinee respond correctly to this item?" The pass mark was calculated by averaging the scores. This pass mark was compared to that created using both the BG and the BLR methods. RESULTS: A paired t-test showed that all examiner groups expected senior residents to get significantly higher percentage of checklist items correct compared to junior residents (all stations p < 0.001). There were no significant differences due to judge type. For senior residents, there were no significant differences between the cut scores determined by the modified Angoff method and the BG/BLR method. For junior residents, the cut scores determined by the modified Angoff method were lower than the cut scores determined by the BG/BLR Method (all p < 0.01). CONCLUSION: The results of this study show that the modified Angoff method is an acceptable method of setting different pass marks for senior and junior residents. The use of this method enables both senior and junior residents to sit the same OSCE, preferable in the regular assessment environment of CBME.

Reclaiming a theoretical orientation to reflection in medical education research: a critical narrative review.

CONTEXT: Reflection and reflective practice have become popular topics of scholarly dialogue in medical education. This popularity has given rise to checklists, portfolios and other tools to inspire and document reflection. We argue that some of the common ways in which reflection has been applied are influenced by broader discourses of assessment and evidence, and divorced from original theories of reflection and reflective practice. METHODS: This paper was developed using a critical narrative approach. First we present two theoretical lenses provided by theories of reflection. Next we present a summary of relevant literature, indexed in PubMed from 2004 to 2014, relating to the application of reflection or reflective practice to undergraduate and postgraduate medical education. We categorise these articles broadly by trends and problematise the trends relative to the two theoretical lenses of reflection. RESULTS: Two relevant theoretical orientations of reflection for medical education are: (i) reflection as epistemology of practice, and (ii) reflection as critical social inquiry. Three prevalent trends in the application of reflection to medical education are: (i) utilitarian applications of reflection; (ii) a focus on the self as the object of reflection, and (iii) reflection and assessment. These trends align with dominant epistemological positions in medicine, but not with those that underpin reflection. CONCLUSIONS: We argue for continued theorising of and theoretically informed applications of reflection, drawing upon epistemologies of practice and critical reflection as critical social inquiry. These directions offer medical education research broad and deep potential in theories of reflection, particularly in relation to knowledge creation within uncertain and complex situations, and challenging of dominant discourses and structures. Future work could explore how dominant epistemological positions and discourses in medicine influence theories from other disciplines when these theories are deployed in medical education.

Seeking inclusion in an exclusive process: discourses of medical school student selection.

CONTEXT: Calls to increase medical class representativeness to better reflect the diversity of society represent a growing international trend. There is an inherent tension between these calls and competitive student selection processes driven by academic achievement. How is this tension manifested? METHODS: Our three-phase interdisciplinary research programme focused on the discourses of excellence, equity and diversity in the medical school selection process, as conveyed by key stakeholders: (i) institutions and regulatory bodies (the websites of 17 medical schools and 15 policy documents from national regulatory bodies); (ii) admissions committee members (ACMs) (according to semi-structured interviews [n = 9]), and (iii) successful applicants (according to semi-structured interviews [n = 14]). The work is theoretically situated within the works of Foucault, Bourdieu and Bakhtin. The conceptual framework is supplemented by critical hermeneutics and the performance theories of Goffman. RESULTS: Academic excellence discourses consistently predominate over discourses calling for greater representativeness in medical classes. Policy addressing demographic representativeness in medicine may unwittingly contribute to the reproduction of historical patterns of exclusion of under-represented groups. In ACM selection practices, another discursive tension is exposed as the inherent privilege in the process is marked, challenging the ideal of medicine as a meritocracy. Applicants' representations of self in the 'performance' of interviewing demonstrate implicit recognition of the power inherent in the act of selection and are manifested in the use of explicit strategies to 'fit in'. CONCLUSIONS: How can this critical discourse analysis inform improved inclusiveness in student selection? Policymakers addressing diversity and equity issues in medical school admissions should explicitly recognise the power dynamics at play between the profession and marginalised groups. For greater inclusion and to avoid one authoritative definition of excellence, we suggest a transformative model of faculty development aimed at promoting multiple kinds of excellence. Through this multi-pronged approach, we call for the profession to courageously confront the cherished notion of the medical meritocracy in order to avoid unwanted aspects of elitism.

Sea monsters & whirlpools: Navigating between examination and reflection in medical education.

The 16th International Ottawa Conference/Canadian Conference on Medical Education (2014) featured a keynote deconstructing the rising discourse of competence-as-reflection in medical education. This paper, an elaborated version of the presentation, is an investigation into the theoretical roots of the diverse forms of reflective practice that are being employed by medical educators. It also raises questions about the degree to which any of these practices is compatible with assessment.

Conceptual and practical challenges in the assessment of physician competencies.

Abstract The shift to using outcomes-based competency frameworks in medical education in many countries around the world requires educators to find ways to assess multiple competencies. Contemporary medical educators recognize that a competent trainee not only needs sound biomedical knowledge and technical skills, they also need to be able to communicate, collaborate and behave in a professional manner. This paper discusses methodological challenges of assessment with a particular focus on the CanMEDS Roles. The paper argues that the psychometric measures that have been the mainstay of assessment practices for the past half-century, while still valuable and necessary, are not sufficient for a competency-oriented assessment environment. New assessment approaches, particularly ones from the social sciences, are required to be able to assess non-Medical Expert (Intrinsic) roles that are situated and context-bound. Realist and ethnographic methods in particular afford ways to address the challenges of this new assessment. The paper considers the theoretical and practical bases for tools that can more effectively assess non-Medical Expert (Intrinsic) roles.

Assessing competencies using milestones along the way.

This paper presents perspectives and controversies surrounding the use of milestones to assess competency in outcomes-based medical education. Global perspectives (Canada, Europe, and the United States) and developments supporting their rationales are discussed. In Canada, there is a significant movement away from conceptualizing competency based on time, and a move toward demonstration of specific competencies. The success of this movement may require complex (rather than reductionist) milestones that reflect students' progression through complexity and context and a method to narrate their journey. European countries (United Kingdom, France, and Germany) have stressed the complexity associated with time and milestones for medical students to truly achieve competence. To meet the changing demands of medicine, they view time as actually providing students with knowledge and exposure to achieve various milestones. In the United States, milestones are based on sampling throughout professional development to initiate lifelong learning. However, the use of milestones may not imply overall competence (reductionism). Milestones must be developed alongside outcomes-based curriculum with use of faculty and competency committees. The perspectives outlined in this paper underscore emerging challenges for implementing outcomes-based medical education and call for new conceptualizations of competence.

Establishing assessment criteria for clinical reasoning in orthopedic manual physical therapy: a consensus-building study.

OBJECTIVES: Clinical reasoning (CR) represents one of the core components of clinical competence in Orthopaedic Manual Physical Therapy (OMPT). While education standards have been developed to guide curricular design, assessment of CR has not yet been standardized. Without theory-informed and rigorously developed measures, the certification of OMPTs lacks credibility and is less defensible. The purpose of this study was to use a theory-informed approach to generate assessment criteria for developing new assessment tools to evaluate CR in OMPT. METHODS: A list of assessment criteria was generated based on international education standards and multiple theoretical perspectives. A modified Delphi method was used to gain expert consensus on the importance of these assessment criteria for the assessment of CR in OMPT. The OMPTs from 22 countries with experience in assessing CR were invited to participate in three rounds of online questionnaires to rate their level of agreement with these criteria. Responses were tabulated to analyze degree of consensus and internal consistency. RESULTS: Representatives from almost half of the OMPT member organizations (MO) participated in three rounds of the Delphi. High levels of agreement were found among respondents regarding the importance and feasibility of most assessment criteria. There was high internal consistency among items within the proposed item subgroupings. DISCUSSION: A list of assessment criteria has been established that will serve as a framework for developing new assessment tools for CR assessment in OMPT. These criteria will be important for guiding the design of certification processes in OMPT as well as other episodes of CR assessment throughout OMPT training.