To prove or improve? Examining how paradoxical tensions shape evaluation practices in accreditation contexts.

INTRODUCTION: Although programme evaluation is increasingly routinised across the academic health sciences, there is scant research on the factors that shape the scope and quality of evaluation work in health professions education. Our research addresses this gap, by studying how the context in which evaluation is practised influences the type of evaluation that can be conducted. Focusing on the context of accreditation, we critically examine the types of paradoxical tensions that surface as evaluation-leads consider evaluation ideals or best practices in relation to contextual demands associated with accreditation seeking. METHODS: Our methods were qualitative and situated within a critical realist paradigm. Study participants were 29 individuals with roles requiring responsibility and oversight on evaluation work. They worked across 4 regions, within 26 academic health science institutions. Data were collected using semi-structured interviews and analysed using framework and matrix analyses. RESULTS: We identified three overarching themes: (i) absence of collective coherence about evaluation practice, (ii) disempowerment of expertise and (iii) tensions as routine practice. Examples of these latter tensions in evaluation work included (i) resourcing accreditation versus resourcing robust evaluation strategy (performing paradox), (ii) evaluation designs to secure accreditation versus design to spur renewal and transformation (performing-learning paradox) and (iii) public dissemination of evaluation findings versus restricted or selective access (publicising paradox). Sub-themes and illustrative data are presented. DISCUSSION: Our study demonstrates how the high-stakes context of accreditation seeking surfaces tensions that can risk the quality and credibility of evaluation practices. To mitigate these risks, those who commission or execute evaluation work must be able to identify and reconcile these tensions. We propose strategies that may help optimise the quality of evaluation work alongside accreditation-seeking efforts. Critically, our research highlights the limitations of continually positioning evaluation purely as a method versus as a socio-technical practice that is highly vulnerable to contextual influences.

Vitalizing the evaluation of curricular implementation: a framework for attending to the "how and whys" of curriculum evolution.

As curricular reforms are implemented, there is often urgency among scholars to swiftly evaluate curricular outcomes and establish whether desired impacts have been realized. Consequently, many evaluative studies focus on summative program outcomes without accompanying evaluations of implementation. This runs the risk of Type III errors, whereby outcome evaluations rest on unverified assumptions about the appropriate implementation of prescribed curricular activities. Such errors challenge the usefulness of the evaluative studies, casting doubt on accumulated knowledge about curricular innovations, and posing problems for educational systems working to mobilize scarce resources. Unfortunately, however, there is long-standing inattention to the evaluation of implementation in health professions education (HPE). To address this, we propose an accessible framework that provides substantive guidance for evaluative research on implementation of curricular innovations. The Prescribed-Intended-Enacted-Sustainable (PIES) framework that is articulated in this paper, introduces new concepts to HPE-with a view to facilitating more nuanced examination of the evolution of curricula as they are implemented. Critically, the framework is theoretically grounded, integrating evaluation and implementation science as well as education theory. It outlines when, how, and why evaluators need to direct attention to curricular implementation, providing guidance on how programs can map out meaningful evaluative research agendas. Ultimately, this work is intended to support evaluators and educators, seeking to design evaluation studies that provide more faithful, useful representations of the intricacies of curricular change implementation.

Five ways to get a grip on the shortcomings of logic models in program evaluation.

Logic models are perhaps the most widely used tools in program evaluation work. They provide reasonably straightforward, visual illustrations of plausible links between program activities and outcomes. Consequently, they are employed frequently in stakeholder engagement, communication, and evaluation project planning. However, their relative simplicity comes with multiple drawbacks that can compromise the integrity of evaluation studies. In this Black Ice article, we outline key considerations and provide practical strategies that can help those engaged in evaluation work to identify and mitigate some limitations of logic models.

Les modèles logiques sont vraisemblablement les outils d'évaluation de programme les plus utilisés. Ils illustrent visuellement de façon assez simple les liens plausibles entre les activités du programme et les résultats obtenus. Par conséquent, ils sont fréquemment utilisés pour la mobilisation des parties prenantes, la communication et la planification de tels projets. Toutefois, leur relative simplicité s'accompagne de multiples inconvénients qui peuvent compromettre l'intégrité des études d'évaluation. Dans cet article de (la rubrique) Terrain glissant, nous proposons des éléments essentiels et des stratégies pratiques à prendre en considération lorsqu'on entreprend une évaluation pour être en mesure de cibler et de remédier à certaines limites des modèles logiques.

Useful to whom? Evaluation utilisation theory and boundaries for programme evaluation scope.

CONTEXT: Programme evaluation is perpetually mandated in health professions education. Correspondingly, there has been an expansion in prescriptive methodological guides about 'how' to engage in various best practices in evaluation. However, what has gained less attention is an examination of the 'value' that different stakeholders seek to gain from programme evaluation. Evaluation utilisation theory and research can help us understand the diversity in both the driving forces for and the impact of programme evaluation. Awareness of the heterogeneity of evaluation utilisation priorities has implications for evaluation practices, including both methodological choices and understanding of the impact of programme evaluation in our field. METHODS: In this article, I expound on the concept of evaluation utilisation by drawing on evaluation theory and research. 'Evaluation utilisation' refers to the application of programme evaluation processes and findings to influence thinking and action. Herein, different forms of 'evaluation utilisation' (including instrumental, conceptual, process and persuasive (symbolic and legitimative) utilisation) are discussed, as well as the related concept of 'evaluation misuse'. Furthermore, how the prioritisation of different forms of 'evaluation utilisation' can influence the scope and impact of evaluation scholarship are also discussed. CONCLUSIONS: Programme evaluation is a form of inquiry that requires more than the exercise of robust methodological techniques. Rather, it necessitates attention to the, sometimes divergent, priorities of different stakeholder groups. Although there is scant research on evaluation practices in health professions education, evaluation utilisation theory can inform critical examination of evaluation practices and impact in our field. Critically, understanding this body of work can help inform those engaged in evaluation about what they are (or should be) prioritising when they conduct programme evaluation, and better align evaluation methodologies with their scholarly, curricular and administrative intentions. Implications for future research and high-quality, transparent evaluation scholarship are presented.

The state-of-play in physician health systems leadership research.

PURPOSE: This paper aims to review a decade of evidence on physician participation in health system leadership with the view to better understand the current state of scholarship on physician leadership activity in health systems. This includes examining the available evidence on both physicians' experiences of health systems leadership (HSL) and the impact of physician leadership on health system reform. DESIGN/METHODOLOGY/APPROACH: A state-of-the-art review of studies (between 2007 and 2017); 51 papers were identified, analyzed thematically and synthesized narratively. FINDINGS: Six main themes were identified in the literature as follows: (De)motivation for leadership, leadership readiness and career development, work demands and rewards, identity matters: acceptance of self (and other) as leader, leadership processes and relationships across health systems and leadership in relation to health system outcomes. There were seemingly contradictory findings across some studies, pointing to the influence of regional and cultural contextual variation on leadership practices as well entrenched paradoxical tensions in health system organizations. RESEARCH LIMITATIONS/IMPLICATIONS: Future research should examine the influence of varying structural and psychological empowerment on physician leadership practices. Empirical attention to paradoxical tensions (e.g. between empowerment and control) in HSL is needed, with specific attention to questions on how such tensions influence leaders' decision-making about system reform. ORIGINALITY/VALUE: This review provides a broad synthesis of diverse papers about physician participation in health system leadership. Thus, it offers a comprehensive empirical synthesis of contemporary concerns and identifies important avenues for future research.

Is postgraduate leadership education a match for the wicked problems of health systems leadership? A critical systematic review.

PURPOSE: There have been a growing number of leadership education programs for physicians. However, debates about the value and efficacy of leadership education in medicine persist, and there are calls for systematic and critical perspectives on medical leadership development. Here, we review evidence on postgraduate leadership education and discuss findings in relation to contemporary evidence on leadership education and practice. METHOD: We searched multiple databases for papers on postgraduate leadership development programs, published in English between 2007 and 2017. We identified 4,691 papers; 31 papers met the full inclusion criteria. Data regarding curricular content and design, learner demographics, instructional methods, and learning outcomes were abstracted and synthesized. RESULTS: There was modest evidence for effectiveness of programs in influencing knowledge and skills gains in select domains. However, the conceptual underpinnings of the 'leadership' training delivered were often unclear. Contemporary theory and evidence on leadership practice was not widely incorporated in program design. Programs were almost exclusively uni-professional, focused on discrete skill development, and did not address systems-level leadership issues. Broader leadership capacity building strategies were underutilized. A new wave of longitudinal, integrated clinical and leadership programming is observed. CONCLUSIONS: Our findings raise questions about persistent preparation-practice gaps in leadership education in medicine. Leadership education needs to evolve to incorporate broader collective capacity building, as well as evidence-informed strategies for leadership development. Barriers to educational reform need to be identified and addressed as educators work to re-orientate education programs to better prepare budding physician leaders for the challenges of health system leadership.

A mandala of faculty development: using theory-based evaluation to explore contexts, mechanisms and outcomes.

Demonstrating the impact of faculty development, is an increasingly mandated and ever elusive goal. Questions have been raised about the adequacy of current approaches. Here, we integrate realist and theory-driven evaluation approaches, to evaluate an intensive longitudinal program. Our aim is to elucidate how faculty development can work to support a range of outcomes among individuals and sub-systems in the academic health sciences. We conducted retrospective framework analysis of qualitative focus group data gathered from 79 program participants (5 cohorts) over a 10-year period. Additionally, we conducted follow-up interviews with 15 alumni. We represent the interactive relationships among contexts, mechanisms, and outcomes as a "mandala" of faculty development. The mandala illustrates the relationship between the immediate program context, and the broader institutional context of academic health sciences, and identifies relevant change mechanisms. Four primary mechanisms were collaborative-reflection, self-reflection and self-regulation, relationship building, and pedagogical knowledge acquisition. Individual outcomes, including changed teaching practices, are described. Perhaps most interestingly, secondary mechanisms-psychological and structural empowerment-contributed to institutional outcomes through participants' engagement in change leadership in their local contexts. Our theoretically informed evaluation approach models how faculty development, situated in appropriate institutional contexts, can trigger mechanisms that yield a range of benefits for faculty and their institutions. The adopted methods hold potential as a way to demonstrate the often difficult-to-measure outcomes of educational programs, and allow for critical examination as to how and whether faculty development programs can accomplish their espoused goals.

Evidence for curricular and instructional design approaches in undergraduate medical education: An umbrella review.

INTRODUCTION: An umbrella review compiles evidence from multiple reviews into a single accessible document. This umbrella review synthesizes evidence from systematic reviews on curricular and instructional design approaches in undergraduate medical education, focusing on learning outcomes. METHODS: We conducted bibliographic database searches in Medline, EMBASE and ERIC from database inception to May 2013 inclusive, and digital keyword searches of leading medical education journals. We identified 18,470 abstracts; 467 underwent duplicate full-text scrutiny. RESULTS: Thirty-six articles met all eligibility criteria. Articles were abstracted independently by three authors, using a modified Kirkpatrick model for evaluating learning outcomes. Evidence for the effectiveness of diverse educational approaches is reported. DISCUSSION: This review maps out empirical knowledge on the efficacy of a broad range of educational approaches in medical education. Critical knowledge gaps, and lapses in methodological rigour, are discussed, providing valuable insight for future research. The findings call attention to the need for adopting evaluative strategies that explore how contextual variabilities and individual (teacher/learner) differences influence efficacy of educational interventions. Additionally, the results underscore that extant empirical evidence does not always provide unequivocal answers about what approaches are most effective. Educators should incorporate best available empirical knowledge with experiential and contextual knowledge.

Reimagining the self at late-career transitions: how identity threat influences academic physicians' retirement considerations.

PURPOSE: There is scant empirical work exploring academic physicians' psychosocial adjustment during late-career transitions or on the factors that influence their retirement decisions. The authors examine these issues through the lens of sociopsychological identity theory, specifically examining how identity threat influences academic physicians' decisions about retirement. METHOD: Participants were academic physicians at a Canadian medical school and were recruited via e-mail requests for clinical faculty interested in discussing late-career and retirement planning issues. Participants included 15 males and 6 females (N = 21; mean age = 63, standard deviation = 7.54), representing eight specialties (clinical and surgical). Data were collected in October and November 2012 via facilitated focus groups, which were digitally recorded, transcribed verbatim, and anonymized, then analyzed using thematic analysis. RESULTS: Four primary themes were identified: centrality of occupational identity, experiences of identity threat, experiences of aging in an indifferent system, and coping with late-career transitions. Identity threats were manifested in apprehensions about self-esteem after retirement, practice continuity, and clinical competence, as well as in a loss of meaning and belonging. These identity challenges influenced decisions on whether to retire. Organizational and system support was perceived as wanting. Coping strategies included reimagining and revaluing various aspects of the self through assimilating new activities and reprioritizing others. CONCLUSIONS: Identity-related struggles are a significant feature of academic physicians' considerations about late-career transitions. Understanding these challenges, their antecedents, and their consequences can prepare faculty, and their institutions, to better manage late-career transitions. Individual- and institution-level implications are discussed.

Affordances of knowledge translation in medical education: a qualitative exploration of empirical knowledge use among medical educators.

PURPOSE: Little is known about knowledge translation processes within medical education. Specifically, there is scant research on how and whether faculty incorporate empirical medical education knowledge into their educational practices. The authors use the conceptual framework of affordances to examine factors within the medical education practice environment that influence faculty utilization of empirical knowledge. METHOD: In 2012, the authors, using a purposive sampling strategy, recruited medical education leaders in undergraduate medical education from a Canadian university. Recruits all had direct teaching and curricular development roles in either preclinical or clinical courses across the four years of the undergraduate curriculum. Data were collected through individual semistructured interviews on participants' use of empirical evidence, as well as the factors that influence integration of empirical knowledge into practice. Data were analyzed using thematic analysis. RESULTS: Fifteen medical educators participated. The authors identified both constraining and facilitating affordances of empirical medical education knowledge use. Constraining affordances included poor quality and availability of evidence, inadequate knowledge delivery approaches, work and role overload, faculty and student change resistance, and resource limitations. Facilitating affordances included faculty development, peer recommendations, and local involvement in medical education knowledge creation. CONCLUSIONS: Affordances of the medical education practice environment influence empirical knowledge use. Developing strategies for effective knowledge translation thus requires careful assessment of contextual factors that can enable, constrain, or inhibit evidence use. Empirical knowledge use is most likely to occur among medical educators who are afforded rich, facilitative opportunities for participation in creating, seeking, and implementing knowledge.