When Feedback is Not Perceived as Feedback: Challenges for Regulatory Body-Mandated Peer Review.

PURPOSE: Safe and competent patient care depends on physicians recognizing and correcting performance deficiencies. Generating effective insight depends on feedback from credible sources. Unfortunately, physicians often have limited access to meaningful guidance. To facilitate quality improvement, many regulatory authorities have designed peer-facilitated practice enhancement programs. Their mandate to ensure practice quality, however, can create tension between formative intentions and risk (perceived or otherwise) of summative repercussions. This study explored how physicians engage with feedback when required to undergo review. METHOD: Between October 2018 and May 2020, 30 physicians representing various specialties and career stages were interviewed about their experiences with peer review in the context of regulatory body-mandated programs. Twenty had been reviewees and reviewers and, hence, spoke from both vantage points. Interview transcripts were analyzed using a 3-stage coding process informed by constructivist grounded theory. RESULTS: Perceptions about the learning value of mandated peer review were mixed. Most saw value but felt anxiety about being selected due to being wary of regulatory bodies. Recognizing barriers such perceptions could create, reviewers described techniques for optimizing the value of interactions with reviewees. Their strategies aligned well with the R2C2 feedback and coaching model with which they had been trained but did not always overcome reviewees' concerns. Reasons included that most feedback was "validating," aimed at "tweaks" rather than substantial change. CONCLUSIONS: This study establishes an intriguing and challenging paradox: feedback appears often to not be recognized as feedback when it poses no threat, yet feedback that carries such threat is known to be suboptimal for inducing performance improvement. In efforts to reconcile that tension, the authors suggest that peer review for individuals with a high likelihood of strong performance may be more effective if expectations are managed through feedforward rather than feedback.

Identification of outcomes in clinical studies of interventions for venous thromboembolism in non-pregnant adults: A scoping review.

BACKGROUND: The development of a core outcome set (COS), defined as an agreed minimum set of outcome domains that should be measured and reported in all trials of a specific disease, aims to increase the relevance of study findings to stakeholder groups and improve standardization. OBJECTIVES: As the first step in developing a COS for venous thromboembolism (VTE) treatment studies, we aimed to generate an inclusive list of unique outcomes reported in previous VTE treatment studies and classify them into domains and core areas. METHODS: MEDLINE, Embase and CENTRAL were searched for prospective studies reporting on interventions for VTE in non-pregnant adults. Study selection and data extraction were performed in blocks based on publication date, starting with 2015-2020 and subsequent 1-year periods, until no new outcome was identified. Outcomes were classified into domains, which are groups of closely related outcomes, and domains into four core areas including death, pathophysiological manifestations/abnormalities, life impact, and resource use. RESULTS: Of 7100 records identified, 240 publications were included, representing 165 distinct studies. A total of 205 unique outcomes were identified that were grouped into 48 domains; 30 (13%) studies covered at least three core areas; death was included in 102 (43%), pathophysiological manifestations/abnormalities in 218 (91%), life impact in 41 (17%), and resource use in 25 (10%) studies. CONCLUSION: Most VTE treatment studies evaluated pathophysiological features of VTE, but few studies reported outcomes that measured life impact or resource use. The findings will inform next steps in the development of a COS for VTE treatment studies.

A phenomenological exploration of the impact of COVID-19 on the medical education community.

INTRODUCTION: The COVID-19 pandemic has caused unprecedented stress to the medical education community, potentially worsening problems like burnout and work-life imbalance that its members have long been grappling with. However, the collective struggle sparked by the pandemic could generate the critical reflection necessary for transforming professional values and practices for the better. In this hermeneutic phenomenological study, we explore how the community is adapting-and even reconceptualising-their personal and professional roles amidst the COVID-19 crisis. METHOD: Between April and October 2020, we conducted 27 (17F, 10M) semi-structured interviews with medical trainees (8), physicians (8), graduate students (3) and PhD scientists (8) working in medical education in Canada, the United States and Switzerland. Data analysis involved a variety of strategies, including coding for van Manen's four lifeworld existentials, reflexive writing and multiple team meetings. RESULTS: Participants experienced grief related to the loss of long-established personal and professional structures and boundaries, relationships and plans for the future. However, experiences of grief were often conflicting. Some participants also experienced moments of relief, perceiving some losses as metaphorical permissions slips to slow down and focus on their well-being. In turn, many reflected on the opportunity they were being offered to re-imagine the nature of their work. DISCUSSION: Participants' experiences with grief, relief and opportunity resonate with Ratcliffe's account of grief as a process of relearning the world after a significant loss. The dismantling of prior life structures and possibilities incited in participants critical reflection on the nature of the medical education community's professional practices. Participants demonstrated their desire for more flexibility and autonomy in the workplace and a re-adjustment of the values and expectations inherent to their profession. On both individual and systems levels, the community must ensure that long-standing calls for wellness and work-life integration are realised-and persist-after the pandemic is over.

From struggle to opportunity: Reimagining medical education in a pandemic era.

The COVID-19 pandemic has disrupted the international medical education community in unprecedented ways. The restrictions imposed to control the spread of the virus have upended our routines and forced us to reimagine our work structures, educational programming and delivery of patient care in ways that will likely continue to change how we live and work for the foreseeable future. Yet, despite these interruptions, the pandemic has additionally sparked a transformative impulse in some to actively engage in critical introspection around the future of their work, compelling us to consider what changes could (and perhaps should) occur after the pandemic is over. Drawing on key concepts associated with scholar Paulo Freire's critical pedagogy, this paper serves as a call to action, illuminating the critical imaginings that have come out of this collective moment of struggle and instability, suggesting that we can perhaps create a more just, compassionate world even in the wake of extraordinary hardship.

Barriers and facilitators to program directors' use of the medical education literature: a qualitative study.

BACKGROUND: It is unclear how often frontline clinical teachers are using this literature and its evidence base in teaching and assessment. Our study purpose was to examine postgraduate program director perspectives on the utilization and integration of evidence-based medical education literature in their teaching and assessment practices. METHODS: The authors conducted semi-structured telephone interviews with a convenience sample of current and former program directors from across Canada. Interviews were transcribed and analyzed inductively to distil pertinent themes. RESULTS: In 2017, 11 former and current program directors participated in interviews. Major themes uncovered included the desire for time-efficient and easily adaptable teaching and assessment tools. Participants reported insufficient time to examine the medical education literature, and preferred that it be 'synthesized for them'. (i.e., Best evidence guidelines). Participants recognised continuing professional development and peer to peer sharing as useful means of education about evidence-based tools. Barriers to the integration of the literature in practice included inadequate time, lack of financial compensation for teaching and assessment, and the perception that teaching and assessment of trainees was not valued in academic promotion. DISCUSSION: Faculty development offices should consider the time constraints of clinical teachers when planning programming on teaching and assessment. To enhance uptake, medical education publications need to consider approaches that best meet the needs of a targeted audiences, including frontline clinical teachers. This may involve novel methods and formats that render evidence and findings from their studies more easily 'digestible' by clinical teachers to narrow the knowledge to practice gap.

Navigating the uncertainty of health advocacy teaching and evaluation from the trainee's perspective.

BACKGROUND: There may be no competency more shrouded in uncertainty than health advocacy (HA), raising questions about the robustness of advocacy training in postgraduate medical education. By understanding how programs currently train HA, we can identify whether trainees' learning needs are being met. METHODS: From 2017 to 2019, we reviewed curricular documents across nine direct-entry specialties at all Ontario medical schools, comparing content for the HA and communicator roles to delineate role-specific challenges. We then conducted semi-structured interviews with trainees (n = 9) and faculty (n = 6) to review findings and discuss their impact. Data were analyzed using thematic content analysis. RESULTS: Curricular documents revealed vague objectives and ill-defined modes of assessment for both intrinsic roles. This uncertainty was perceived as more problematic for HA, in part because HA seemed both undervalued in, and disconnected from, clinical learning. Trainees felt that the onus was on them to figure out how to develop and demonstrate HA competence, causing many to turn their learning attention elsewhere. DISCUSSION: Lack of curricular focus seems to create the perception that advocacy isn't valuable, deterring trainees-even those keen to become competent advocates-from developing HA skills. Such ambivalence may have troubling downstream effects for both patient care and trainees' professional development.

Wrestling With the Invincibility Myth: Exploring Physicians' Resistance to Wellness and Resilience-Building Interventions.

PURPOSE: Physicians are expected to provide compassionate, error-free care while navigating systemic challenges and organizational demands. Many are burning out. While organizations are scrambling to address the burnout crisis, physicians often resist interventions aimed at enhancing their wellness and building their resilience. The purpose of this research was to empirically study this phenomenon. METHOD: Constructivist grounded theory was used to inform the iterative data collection and analysis process. In spring 2018, 22 faculty physicians working in Canada participated in semistructured interviews to discuss their experiences of wellness and burnout, their perceptions of wellness initiatives, and how their experiences and perceptions influence their uptake of the rapidly proliferating strategies aimed at nurturing their resilience. Themes were identified using constant comparative analysis. RESULTS: Participants suggested that the values of compassion espoused by health care organizations do not extend to physicians, and they described feeling dehumanized by professional values steeped in an invincibility myth in which physicians are expected to be "superhuman" and "sacrifice everything" for medicine. Participants described that professional values and organizational norms impeded work-life balance, hindered personal and professional fulfillment, and discouraged disclosure of struggles. In turn, participants seemed to resist wellness and resilience-building interventions focused on fixing individuals rather than broader systemic, organizational, and professional issues. Participants perceived that efforts aimed at building individual resilience are futile without changes in professional values and sustained organizational support. CONCLUSIONS: Findings suggest that professional and organizational norms and expectations trigger feelings of dehumanization for some physicians. These feelings likely exacerbate burnout and may partly explain physicians' resistance to resilience-building strategies. Mitigating burnout and developing and sustaining a resilient physician workforce will require both individual resistance to problematic professional values and an institutional commitment to creating a culture of compassion for patients and physicians alike.

The Impact of Surgeon Experience on Script Concordance Test Scoring.

OBJECTIVE: The Script Concordance Test (SCT) is a test of clinical decision-making that relies on an expert panel to create its scoring key. Existing literature demonstrates the value of specialty-specific experts, but the effect of experience among the expert panel is unknown. The purpose of this study was to explore the role of surgeon experience in SCT scoring. DESIGN: An SCT was administered to 29 general surgery residents and 14 staff surgeons. Staff surgeons were stratified as either junior or senior experts based on years since completing residency training (<15 versus >25 years). The SCT was scored using the full expert panel, the senior panel, the junior panel, and a subgroup junior panel in practice <5 years. A one-way ANOVA was used to compare the scores of first (R1) and fifth (R5) year residents using each scoring scheme. Cognitive interviews were analyzed for differences between junior and senior expert panelist responses. RESULTS: There was no statistically significant difference between the mean score of six R1s and five R5s using the full expert panel (R1 69.08 versus R5 67.06, F1,9 = 0.10, P = 0.76), the junior panel (R1 66.73 versus R5 62.50, F1,9 = 0.35, P = 0.57), or the subgroup panel in practice <5 years (R1 61.07 versus R5 58.79, F1,9 = 0.18, P = 0.75). However, the average score of R1s was significantly lower than R5s when using the senior faculty panel (R1 52.04 versus R5 63.26, F1,9 = 26.90, P = 0.001). Cognitive interview data suggests that some responses of junior experts demonstrate less confidence than those of senior experts. CONCLUSIONS: SCT scores are significantly affected by the responses of the expert panel. Expert differences between first and fifth year residents were only demonstrated when using an expert panel consisting of senior faculty members. Confidence may play a role in the response selections of junior experts. When constructing an SCT expert panel, consideration must be given to the experience of panel members.

Are raters influenced by prior information about a learner? A review of assimilation and contrast effects in assessment.

Understanding which factors can impact rater judgments in assessments is important to ensure quality ratings. One such factor is whether prior performance information (PPI) about learners influences subsequent decision making. The information can be acquired directly, when the rater sees the same learner, or different learners over multiple performances, or indirectly, when the rater is provided with external information about the same learner prior to rating a performance (i.e., learner handover). The purpose of this narrative review was to summarize and highlight key concepts from multiple disciplines regarding the influence of PPI on subsequent ratings, discuss implications for assessment and provide a common conceptualization to inform research. Key findings include (a) assimilation (rater judgments are biased towards the PPI) occurs with indirect PPI and contrast (rater judgments are biased away from the PPI) with direct PPI; (b) negative PPI appears to have a greater effect than positive PPI; (c) when viewing multiple performances, context effects of indirect PPI appear to diminish over time; and (d) context effects may occur with any level of target performance. Furthermore, some raters are not susceptible to context effects, but it is unclear what factors are predictive. Rater expertise and training do not consistently reduce effects. Making raters more accountable, providing specific standards and reducing rater cognitive load may reduce context effects. Theoretical explanations for these findings will be discussed.

Implementation and evaluation of a curriculum on the assessment and treatment of disruptive behaviour disorders.

Disruptive behaviour disorders (DBDs)-which can include or be comorbid with disorders such as attention-deficit hyperactivity disorder, oppositional defiant disorder, conduct disorder and disruptive mood dysregulation disorder-are commonly seen in paediatric practice. Given increases in the prescribing of atypical antipsychotics for children and youth, it is imperative that paediatric trainees in Canada receive adequate education on the optimal treatment of DBDs. We describe the development, dissemination, and evaluation of a novel paediatric resident curriculum for the assessment and treatment of DBDs in children and adolescents. Pre-post-evaluation of the curriculum showed improved knowledge in participants.

How do cognitive processes influence script concordance test responses?

INTRODUCTION: The script concordance test (SCT) is a test of clinical decision-making (CDM) that compares the thought process of learners to that of experts to determine to what extent their cognitive 'scripts' align. Without understanding test-takers' cognitive process, however, it is unclear what influences their responses. The objective of this study was to gather response process validity evidence by studying the cognitive process of test-takers to determine whether the SCT tests CDM and what cognitive processes may influence SCT responses. METHODS: Cases from an SCT used in a national validation study were administered and semi-structured cognitive interviews were conducted with ten residents and five staff surgeons. A retrospective verbal probing technique was used. Data was independently analysed and coded by two analysts. Themes were identified as factors that influence SCT responses during the cognitive interview. RESULTS: Cognitive interviews demonstrated variability in CDM among test-takers. Consistent with dual process theory, test-takers relied on scripts formed through past experiences, when available, to make decisions and used conscious deliberation in the absence of experience. However, test-takers' response process was also influenced by their comprehension of specific terms, desire for additional information, disagreement with the planned management, underlying knowledge gaps and desire to demonstrate confidence or humility. CONCLUSION: The rationale behind SCT answers may be influenced by comprehension, underlying knowledge and social desirability in addition to formed scripts and/or conscious deliberation. Having test-takers verbalise their rationale for responses provides a depth of assessment that is otherwise lost in the SCT's current format. With the improved ability to standardise CDM assessment using the SCT, consideration of test-makers improving the SCT construction process and combining the SCT question format with verbal responses may improve the use of the SCT for CDM assessment.

The cognitive process of test takers when using the script concordance test rating scale.

CONTEXT: Clinical decision making (CDM) skills are important to learn and assess in order to establish competence in trainees. A common tool for assessing CDM is the script concordance test (SCT), which asks test takers to indicate how a new clinical finding influences a proposed plan using a Likert-type scale. Most criticisms of the SCT relate to its rating scale but are largely theoretical. The cognitive process of test takers when selecting their responses using the SCT rating scale remains understudied, but is essential to gathering validity evidence for use of the SCT in CDM assessment. METHODS: Cases from an SCT used in a national validation study were administered to 29 residents and 14 staff surgeons. Semi-structured cognitive interviews were then conducted with 10 residents and five staff surgeons based on the SCT results. Cognitive interview data were independently coded by two data analysts, who specifically sought to elucidate how participants mapped their internally generated responses to any of the rating scale options. RESULTS: Five major issues were identified with the response matching cognitive process: (a) the meaning of the '0' response option; (b) which response corresponds to agreement with the planned management; (c) the rationale for picking '±1' versus '±2'; (d) which response indicates the desire to undertake the planned management plus an additional procedure, and (e) the influence of time on response selection. CONCLUSIONS: Studying how test takers (experts and trainees) interpret the SCT rating scale has revealed several issues related to inconsistent and unintended use. Revising the scale to address the variety of interpretations could help to improve the response process validity of the SCT and therefore improve the SCT's ability to be used in CDM skills assessments.

The Influence of Prior Performance Information on Ratings of Current Performance and Implications for Learner Handover: A Scoping Review.

PURPOSE: Learner handover (LH) is the sharing of information about trainees between faculty supervisors. This scoping review aimed to summarize key concepts across disciplines surrounding the influence of prior performance information (PPI) on current performance ratings and implications for LH in medical education. METHOD: The authors used the Arksey and O'Malley framework to systematically select and summarize the literature. Cross-disciplinary searches were conducted in six databases in 2017-2018 for articles published after 1969. To represent PPI relevant to LH in medical education, eligible studies included within-subject indirect PPI for work-type performance and rating of an individual current performance. Quantitative and thematic analyses were conducted. RESULTS: Of 24,442 records identified through database searches and 807 through other searches, 23 articles containing 24 studies were included. Twenty-two studies (92%) reported an assimilation effect (current ratings were biased toward the direction of the PPI). Factors modifying the effect of PPI were observed, with larger effects for highly polarized PPI, negative (vs positive) PPI, and early (vs subsequent) performances. Specific standards, rater motivation, and certain rater characteristics mitigated context effects, whereas increased rater processing demands heightened them. Mixed effects were seen with nature of the performance and with rater expertise and training. CONCLUSIONS: PPI appears likely to influence ratings of current performance, and an assimilation effect is seen with indirect PPI. Whether these findings generalize to medical education is unknown, but they should be considered by educators wanting to implement LH. Future studies should explore PPI in medical education contexts and real-world settings.