Not in the file: How competency committees work with undocumented contributions.

INTRODUCTION: Competence committees (CCs) centre their work around documentation of trainees' performance; undocumented contributions (i.e. informal, unrecorded material like personal judgements, experiential anecdotes and contextual information) evoke suspicion even though they may play a role in decision making. This qualitative multiple case study incorporates insights from a social practice perspective on writing to examine the use of undocumented contributions by the CCs of two large post-graduate training programmes, one in a more procedural (MP) speciality and the other in a less procedural (LP) one. METHODS: Data were collected via observations of meetings and semi-structured interviews with CC members. In the analysis, conversations were organised into triptychs of lead-up, undocumented contribution(s), and follow-up. We then created thick descriptions around the undocumented contributions, drawing on conversational context and interview data to assign possible motivations and significance. RESULTS: We found no instances in which undocumented contributions superseded the contents of a trainee's file or stood in for missing documentation. The number of undocumented contributions varied between the MP CC (six instances over two meetings) and the LP CC (22 instances over three meetings). MP CC discussions emphasised Entrustable Professional Activity (EPA) observations, whereas LP CC members paid more attention to narrative data. The divergent orientations of the CCs-adding an 'advis[ing]/guid[ing]' role versus focusing simply on evaluation-offers the most compelling explanation. In lead-ups, undocumented contributions were prompted by missing and flawed documentation, conflicting evidence and documentation at odds with members' perceptions. Recognising other 'red flags' in documentation often required professional experience. In follow-ups, purposes served by undocumented contributions varied with context and were difficult to generalise; we, therefore, provide deeper analysis of two vignettes to illustrate. CONCLUSIONS: Our data suggest undocumented contributions often serve best efforts to ground decisions in documentation. We would encourage CC practices and policies be rooted in more nuanced approaches to documentation.

Dual purposes by design: exploring alignment between residents' and academic advisors' documents in a longitudinal program.

Longitudinal academic advising (AA) and coaching programs are increasingly implemented in competency based medical education (CBME) to help residents reflect and act on the voluminous assessment data they receive. Documents created by residents for purposes of reflection are often used for a second, summative purpose-to help competence committees make decisions-which may be problematic. Using inductive, thematic analysis we analyzed written comments generated by 21 resident-AA dyads in one large internal medicine program who met over a 2 year period to determine what residents write when asked to reflect, how this aligns with what the AAs report, and what changes occur over time (total 109 resident self-reflections and 105 AA reports). Residents commented more on their developing autonomy, progress and improvement than AAs, who commented far more on performance measures. Over time, residents' writing shifted away from intrinsic roles, patient care and improvement towards what AAs focused on, including getting EPAs (entrustable professional activities), studying and exams. For EPAs, the emphasis was on getting sufficient numbers rather than reflecting on what residents were learning. Our findings challenge the practice of dual-purposing documents, by questioning the blurring of formative and summative intent, the structure of forms and their multiple conflicting purposes, and assumptions about the advising relationship over time. Our study suggests a need to re-evaluate how reflective documents are used in CBME programs. Further research should explore whether and how documentation can best be used to support resident growth and development.

Goal co-construction and dialogue in an internal medicine longitudinal coaching programme.

BACKGROUND: Longitudinal coaching in residency programmes is becoming commonplace and requires iterative and collaborative discussions between coach and resident, with the shared development of goals. However, little is known about how goal development unfolds within coaching conversations over time and the effects these conversations have. We therefore built on current coaching theory by analysing goal development dialogues within resident and faculty coaching relationships. METHODS: This was a qualitative study using interpretive description methodology. Eight internal medicine coach-resident dyads consented to audiotaping coaching meetings over a 1-year period. Transcripts from meetings and individual exit interviews were analysed thematically using goal co-construction as a sensitising concept. RESULTS: Two themes were developed: (i) The content of goals discussed in coaching meetings focused on how to be a resident, with little discussion around challenges in direct patient care, and (ii) co-construction mainly occurred in how to meet goals, rather than in prioritising goals or co-constructing new goals. CONCLUSIONS: In analysing goal development in the coach-resident relationships, conversations focused mainly around how to manage as a resident rather than how to improve direct patient care. This may be because academic coaching provides space separate from clinical work to focus on the stage-specific professional identity development of a resident. Going forward, focus should be on how to optimise longitudinal coaching conversations to ensure co-regulation and reflection on both clinical competencies and professional identity formation.

A systematic review of evidence-based practices for clinical education and health care delivery in the clinical teaching unit.

BACKGROUND: The clinical teaching unit is a widespread clinical training model that requires reform to prepare physicians for practice in the 21st century. In this systematic review, we aimed to identify evidence-based practices in internal medicine clinical teaching units that contribute to improved clinical education and health care delivery. METHODS: We searched several databases from 1993 until Apr. 5, 2021, to identify published studies in inpatient clinical teaching units that involved medical trainees and reported outcomes related to trainee education or health care delivery. We identified emergent themes using a narrative approach and determined confidence in review findings using the Grading of Recommendations Assessment, Development and Evaluation Confidence in the Evidence from Reviews of Qualitative Research (GRADE-CERQual) methodology. RESULTS: We included 107 studies of internal medicine clinical teaching units, of which 93 (87%) were conducted in North America. Surveys (n = 31, 29%), trials (n = 17, 16%) and narrative studies (n = 15, 14%) were the most prevalent study designs. Practices identified as contributing to improved clinical education or health care delivery included purposeful rounding (high confidence), bedside rounding (moderate confidence), resource stewardship interventions (high confidence), interprofessional rounds (moderate confidence), geographic wards (moderate confidence), allocating more trainee time to patient care or educational activities (moderate confidence), "drip" continuous models of admission (moderate confidence), limiting duty hours (moderate confidence) and limiting clinical workload (moderate confidence). INTERPRETATION: In this review, we identified several evidence-based practices that may contribute to improved educational and health care outcomes in clinical teaching unit settings. These findings may offer guidance for policies, resource allocation and staffing of teaching hospitals.

A qualitative study of medical students' perceptions of resident feedback.

CONTEXT: Residents play a pivotal role in medical students' clinical education. From a feedback lens, the near-peer relationship between student and resident holds the potential to foster an educational alliance that could influence learning. We undertook the current qualitative study to explore medical students' perceptions of feedback experiences with residents, addressing when, how and why (and conversely when not and why not) resident feedback plays a role in their clinical education. METHODS: Our methodology was qualitative interpretive description, informed by phenomenology. We conducted 24 semi-structured interviews with third and fourth year medical students at one institution. The interviews aimed to foster rich discussion about students' feedback experiences with residents during clinical rotations. Data collection and analysis proceeded iteratively. Initial interviews were independently open-coded by three investigators and then collaboratively refined. Codes were applied to subsequent interviews, and new codes were developed. During the final stages of analysis, we organised our themes by drawing on a sociocultural perspective to examine students' perceptions of relationship-building with residents and when and how this influenced feedback and learning. RESULTS: From the students' perspectives, when residents contributed to building interpersonal relationships with students, this in turn influenced students' receptivity to both encouraging and constructive feedback conversations. In the context of resident-student relationships that were perceived as supportive, resident feedback influenced how students approached learning and working in the clinical environment, as well as students' visions of their future selves. In unsupportive relationships, students were less inclined to engage in feedback with residents and students noted resident behaviours that they wanted to avoid in themselves. CONCLUSION: Residents are uniquely positioned to create a strong educational alliance with students in which feedback conversations can flourish. Focusing educational efforts on resident feedback conversations has the potential to significantly impact the feedback culture of our clinical environments.

Supervising the senior medical resident: Entrusting the role, supporting the tasks.

INTRODUCTION: Postgraduate competency-based medical education has been implemented with programmatic assessment that relies on entrustment-based ratings. Yet, in less procedurally oriented specialties such as internal medicine, the relationship between entrustment and supervision remains unclear. We undertook the current study to address how internal medicine supervisors conceptualise entrusting senior medical residents while supervising them on the acute care wards. METHODS: Guided by constructivist grounded theory, we interviewed 19 physicians who regularly supervised senior internal medicine residents on inpatient wards at three Canadian universities. We developed a theoretical model through iterative cycles of data collection and analysis using a constant comparative process. RESULTS: On the internal medicine ward, the senior resident role is viewed as a fundamentally managerial and rudimentary version of the supervisor's role. Supervisors come to trust their residents in the senior role through an early 'hands-on' period of assessment followed by a gradual withdrawal of support to promote independence. When considering entrustment, supervisors focused on entrusting a particular scope of the senior resident role as opposed to entrustment of individual tasks. Irrespective of the scope of the role that was entrusted, supervisors at times stepped in and stepped back to support specific tasks. CONCLUSION: Supervisors' stepping in and stepping back to support individual tasks on the acute care ward has an inconsistent relationship to their entrustment of the resident with a particular scope of the senior resident role. In this context, entrustment-based assessment would need to capture more of the holistic perspective of the supervisor's entrustment of the senior resident role. Understanding the dance of supervision, from relatively static overall support of the resident in their role, to fluidly stepping in and out for specific patient care tasks, allows us insight into the affordances of the supervisory relationship and how it may be leveraged for assessment.

Examining the validity argument for the Ottawa Surgical Competency Operating Room Evaluation (OSCORE): a systematic review and narrative synthesis.

The Ottawa Surgical Competency Operating Room Evaluation (OSCORE) is an assessment tool that has gained prominence in postgraduate competency-based training programs. We undertook a systematic review and narrative synthesis to articulate the underlying validity argument in support of this tool. Although originally developed to assess readiness for independent performance of a procedure, contemporary implementation includes using the OSCORE for entrustment supervision decisions. We used systematic review methodology to search, identify, appraise and abstract relevant articles from 2005 to September 2020, across MEDLINE, EMBASE and Google Scholar databases. Nineteen original, English-language, quantitative or qualitative articles addressing the use of the OSCORE for health professionals' assessment were included. We organized and synthesized the validity evidence according to Kane's framework, articulating the validity argument and identifying evidence gaps. We demonstrate a reasonable validity argument for the OSCORE in surgical specialties, based on assessing surgical competence as readiness for independent performance for a given procedure, which relates to ad hoc, retrospective, entrustment supervision decisions. The scoring, generalization and extrapolation inferences are well-supported. However, there is a notable lack of implications evidence focused on the impact of the OSCORE on summative decision-making within surgical training programs. In non-surgical specialties, the interpretation/use argument for the OSCORE has not been clearly articulated. The OSCORE has been reduced to a single-item global rating scale, and there is limited validity evidence to support its use in workplace-based assessment. Widespread adoption of the OSCORE must be informed by concurrent data collection in more diverse settings and specialties.

The nature of learning from simulation: Now I know it, now I'll do it, I'll work on that.

CONTEXT: Ongoing learning in complex clinical environments requires health professionals to assess their own performance, manage their learning, and modify their practices based on self-monitored progress. Self-regulated learning (SRL) theory suggests that although learners may be capable of such learning, they often need guidance to enact it effectively. Debriefings following simulation may be an ideal time to support learners' use of SRL in targeted areas, but the extent to which they are optimally fostering these practices has not been examined. METHODS: A qualitative study informed by grounded theory methodology was conducted in the context of three interprofessional in situ trauma simulations at our level 1 trauma centre. A total of 18 participants were interviewed both immediately and 5-6 weeks after the simulation experience. Transcripts were analysed using an iterative constant comparative approach to explore concepts and themes regarding the nature of learning from and after simulation. RESULTS: During initial interviews, there were many examples of acquired content knowledge and straightforward practice changes that might not require ongoing SRL to enact well in practice. However, even for skills identified as needing to be 'worked on,' SRL strategies were lacking. At follow-up interviews, some participants had evolved more specific learning goals and rudimentary plans for implementation and improvement, but suggested this was prompted by the study interview questions rather than the simulation debriefing itself. CONCLUSIONS: Overall, participants did not engage in fulsome development of SRL plans based on the simulation and debriefing; however, there were elements of SRL present, particularly after participants were given time to reflect on the interview questions and their own goals. This suggests that simulation training can support the use of SRL. However, debriefing approaches might be better optimised to take full advantage of the opportunity to encourage and foster SRL in practice after the simulation is over.

Necessary but not sufficient: identifying conditions for effective feedback during internal medicine residents' clinical education.

Competency-based medical education and programmatic assessment intend to increase the opportunities for meaningful feedback, yet these conversations remain elusive. By comparing resident and faculty perceptions of feedback opportunities within one internal medicine residency training program, we sought to understand whether and how principles underlying meaningful feedback could be supported or constrained across a variety of feedback opportunities. Using case-study qualitative methodology, interviews and focus groups were conducted to explore 19 internal medicine residents' and 7 faculty members' perceptions of feedback across a variety of feedback opportunities: coaching, mini-CEXs, in-training evaluation reports and routine clinical supervision. Our data analysis moved iteratively between developing conceptual understandings and fine-grained analyses, while attending to both deductive and inductive analysis. Our results suggest that all feedback opportunities, including those created through formalized assessments, can foster meaningful feedback if faculty establish a trusting relationship with the resident, base their feedback on direct observation and support resident learning. However, formalized assessments were often perceived as inhibiting the conditions for meaningful feedback. A coaching program provided a context in which meaningful feedback could arise, in part because faculty were supported in shifting their focus from patient to resident. Meaningful feedback in clinical education may be fostered across a variety of feedback opportunities, however, it is often constrained by assessment. We must consider whether increasing the frequency of formative assessments may inhibit efforts to improve our feedback cultures while, in contrast, freeing up faculty to focus on supporting resident learning could improve these cultures.

Entrustment Ratings in Internal Medicine Training: Capturing Meaningful Supervision Decisions or Just Another Rating?

The implementation of Entrustable Professional Activities has led to the simultaneous development of assessment based on a supervisor's entrustment of a learner to perform these activities without supervision. While entrustment may be intuitive when we consider the direct observation of a procedural task, the current implementation of rating scales for internal medicine's non-procedural tasks, based on entrustability, may not translate into meaningful learner assessment. In these Perspectives, we outline a number of potential concerns with ad hoc entrustability assessments in internal medicine post-graduate training: differences in the scope of procedural vs. non-procedural tasks, acknowledgement of the type of clinical oversight common within internal medicine, and the limitations of entrustment language. We point towards potential directions for inquiry that would require us to clarify the purpose of the entrustability assessment, reconsider each of the fundamental concepts of entrustment in internal medicine supervision and explore the use of descriptive rather than numeric assessment approaches.

How well is each learner learning? Validity investigation of a learning curve-based assessment approach for ECG interpretation.

Learning curves can support a competency-based approach to assessment for learning. When interpreting repeated assessment data displayed as learning curves, a key assessment question is: "How well is each learner learning?" We outline the validity argument and investigation relevant to this question, for a computer-based repeated assessment of competence in electrocardiogram (ECG) interpretation. We developed an on-line ECG learning program based on 292 anonymized ECGs collected from an electronic patient database. After diagnosing each ECG, participants received feedback including the computer interpretation, cardiologist's annotation, and correct diagnosis. In 2015, participants from a single institution, across a range of ECG skill levels, diagnosed at least 60 ECGs. We planned, collected and evaluated validity evidence under each inference of Kane's validity framework. For Scoring, three cardiologists' kappa for agreement on correct diagnosis was 0.92. There was a range of ECG difficulty across and within each diagnostic category. For Generalization, appropriate sampling was reflected in the inclusion of a typical clinical base rate of 39% normal ECGs. Applying generalizability theory presented unique challenges. Under the Extrapolation inference, group learning curves demonstrated expert-novice differences, performance increased with practice and the incremental phase of the learning curve reflected ongoing, effortful learning. A minority of learners had atypical learning curves. We did not collect Implications evidence. Our results support a preliminary validity argument for a learning curve assessment approach for repeated ECG interpretation with deliberate and mixed practice. This approach holds promise for providing educators and researchers, in collaboration with their learners, with deeper insights into how well each learner is learning.

Integrated virtual and cadaveric dissection laboratories enhance first year medical students' anatomy experience: a pilot study.

BACKGROUND: Radiology integration into medical anatomy courses is well established, but there is a paucity of literature on integrating virtual dissection into cadaveric dissection laboratories. Virtual dissection is the digital dissection of medical images on touchscreen anatomy visualization tables. The purpose of this pilot study was to investigate the feasibility of integrating virtual dissection into a first-year medical cadaver-based anatomy course and to assess students' overall attitude towards this new technology. METHODS: All students in first-year medicine at a single medical school participated in this study (n = 292). Six virtual dissection laboratories, which focused on normal anatomy, were developed and integrated into a cadaver-based anatomy course. The virtual dissection table (VDT) was also integrated into the final anatomy spot exam. Following the course, students completed a short evidence-informed survey which was developed using a theoretical framework for curriculum evaluation. Numerical data were tabulated, and qualitative content analysis was performed on students' unstructured comments. RESULTS: The survey response rate was 69.2% (n = 202/292). Most (78.7%) students reported that virtual dissection enhanced their understanding of the cadaveric anatomy and the clinical applications of anatomy. Most (73.8%) students also felt that the VDT was an effective use of the laboratory time. Thirteen narrative comments were collected, most of which (61.5%) identified strengths of the curriculum. CONCLUSIONS: In this pilot study, students perceived that their learning was enhanced when virtual dissection was combined with a cadaver-based anatomy laboratory. This study demonstrates that there is potential for virtual dissection to augment cadaveric dissection in medical education.

Beyond hands-on and hands-off: supervisory approaches and entrustment on the inpatient ward.

CONTEXT: The concept of entrustment has garnered significant attention in medical specialties, despite variability in supervision styles and entrustment decisions. There is a need to further study the enactment of supervision on inpatient wards to inform competency-based assessment design. METHODS: Attending physicians, while supervising on clinical teaching inpatient wards, were invited to describe a recent moment of enacting supervision with an internal medicine resident. Constructivist grounded theory guided data collection and analysis. Interview transcripts were analysed in iterative cycles to inform data collection. Constant comparison was used to build a theory of supervision from the identified themes. RESULTS: In 2016-2017, 23 supervisors from two Canadian universities with supervision reputations ranging from very involved to less involved participated in one or two interviews (total: 28). Supervisors were not easily dichotomised into styles based on behaviour because all used similar oversight strategies. Supervisors described adjusting between 'hands-on' (e.g. detail oriented) and 'hands-off' (e.g. less visible on ward) styles depending on the context. All also contended with the competing roles of clinical teacher and care provider. Supervisors made a distinction between the terms `entrust' and `trust', and did not grant complete entrustment to senior residents. CONCLUSIONS: We propose that a supervisor's perceived responsibility for the ward underlies adjustments between 'hands-on' (i.e. personal ward responsibility) and 'hands-off' (i.e. shared ward responsibility) styles. Our approaches to clinical supervision model combines this responsibility tension with the tension between patient care and teaching to illustrate four supervisory approaches, each with unique priorities influencing entrustment. Given the fluidity in supervision, documenting changes in oversight strategies, rather than absolute levels of entrustment, may be more informative for assessment purposes. Research is needed to determine if there is sufficient association between the supervision provided, the entrustment decision made and the supervisor's trust in a trainee to use these as proxies in assessing a trainee's competence.

Development and Evaluation of a Competency-Based Anatomy Rotation for Diagnostic Radiology Residents During Internship Year: A Canadian Experience.

RATIONALE AND AIM: As medical schools reduce the hours of anatomy teaching, residents in anatomy-intensive residency programs like radiology must independently acquire the anatomy knowledge needed to achieve competency. The purpose of this study was to develop and evaluate a 4-week competency-based self-directed anatomy rotation for junior residents. METHODS: Seven post-graduate year 1 (PGY-1) radiology residents completed a 4-week rotation of radiologic anatomy. The objectives were developed from standards, senior residents, and expert opinion, and the competency-based curriculum included self-directed modules. Pre-course and post-course tests were administered and test scores were compared using an unpaired t test. In addition, PGY-1 residents completed a course evaluation and survey regarding their anatomy knowledge and anatomy exposure prior to completing the course. RESULTS: Out of the 25 points available, the average pre-test score was 10.79 ± 2.78 (range 8-16.5), and the average post-test score was 21.64 ± 2.23 (range 18.5-25). This difference was statistically significant (P < .0001). The PGY-1 residents reported receiving < 10% of dedicated radiologic anatomy teaching prior to residency and felt unprepared for the anatomy required in residency. Overall, residents felt more confident in looking at images after completing the self-directed radiologic anatomy course. CONCLUSION: This study demonstrates the feasibility of creating a self-directed course for radiology residents that significant improves their anatomy knowledge. Given the trend in medical undergraduate education away from dedicated anatomy teaching, residency programs should consider addressing anatomy education more formally for junior residents to ensure that trainees receive the foundational knowledge required for residency.

Staging a performance: learners' perceptions about direct observation during residency.

CONTEXT: Evidence strongly supports that direct observation is a valid and reliable assessment tool; support for its impact on learning is less compelling, and we know that some learners are ambivalent about being observed. However, learners' perceptions about the impact of direct observation on their learning and professional development remain underexplored. To promote learning, we need to understand what makes direct observation valuable for learners. METHODS: Informed by constructivist grounded theory, we interviewed 22 learners about their observation experiences. Data collection and analysis occurred iteratively; themes were identified using constant comparative analysis. RESULTS: Direct observation was widely endorsed as an important educational strategy, albeit one that created significant anxiety. Opaque expectations exacerbated participants' discomfort, and participants described that being observed felt like being assessed. Consequently, participants exchanged their 'usual' practice for a 'textbook' approach; alterations to performance generated uncertainty about their role, and raised questions about whether observers saw an authentic portrayal of their knowledge and skill. CONCLUSION: An 'observer effect' may partly explain learners' ambivalence about direct observation; being observed seemed to magnify learners' role ambiguity, intensify their tensions around professional development and raise questions about the credibility of feedback. In turn, an observer effect may impact learners' receptivity to feedback and may explain, in part, learners' perceptions that useful feedback is scant. For direct observation to be valuable, educators must be explicit about expectations, and they must be aware that how learners perform in the presence of an observer may not reflect what they do as independent practitioners. To nurture learners' professional development, educators must create a culture of observation-based coaching that is divorced from assessment and is tailored to developing learners' identities as practitioners of both the art and the science of medicine.

Investigating conditions for meaningful feedback in the context of an evidence-based feedback programme.

CONTEXT: We developed, implemented and evaluated an evidence-based programme of feedback designed to address limitations identified in the current literature. OBJECTIVES: We sought to advance understanding about how and why feedback processes might be more effective in clinical education. METHODS: Three faculty members and nine first-year internal medicine residents participated in the pilot programme. To counter challenges identified in the literature, feedback was based on direct observation, grounded in longitudinal faculty-resident relationships, and devoid of summative assessment. We used a qualitative case study design to address three research questions: (i) What benefits did the participants describe? (ii) What elements of the programme facilitated these benefits? (iii) What were the limitations and challenges of the programme? Collected data included audiotapes of interactions between faculty members and residents, field notes written during observations, and semi-structured interviews and focus groups with resident participants. Data analysis moved cyclically and iteratively through inductive and deductive analysis. RESULTS: Residents described benefits relating to their ways of working (clinical skills), ways of learning (accountability for learning) and ways of feeling (emotional well-being). According to participants, specific elements of the programme that achieved these benefits included the direct observation of authentic clinical work, the longitudinal relationship with a faculty member and the emergence of feedback as a conversation between the faculty member and learner. CONCLUSIONS: We conclude that the conditions established within our pilot feedback programme influenced the learning culture for first-year internal medicine residents by grounding direct observation in authentic clinical work and setting the observations in the context of a longitudinal, non-assessment-based relationship between a faculty member and resident. These conditions appeared to influence residents' participation in the feedback process, their ways of approaching their daily clinical work, their emotional well-being and their engagement in their own learning.

'Sometimes the work just needs to be done': socio-cultural influences on direct observation in medical training.

CONTEXT: Direct observation promises to strengthen both coaching and assessment, and calls for its increased use in medical training abound. Despite its apparent potential, the uptake of direct observation in medical training remains surprisingly limited outside the formal assessment setting. The limited uptake of observation raises questions about cultural barriers to its use. In this study, we explore the influence of professional culture on the use of direct observation within medical training. METHODS: Using a constructivist grounded theory approach, we interviewed 22 residents or fellows (10 male, 12 female) about their experiences of being observed during training. Participants represented a range of specialties and training levels. Data collection and analysis were conducted iteratively. Themes were identified using constant comparative analysis. RESULTS: Observation was used selectively; specialties tended to observe the clinical acts that they valued most. Despite these differences, we found two cultural values that consistently challenged the ready implementation of direct observation across specialties: (i) autonomy in learning and (ii) efficiency in health care provision. Furthermore, we found that direct observation was a primarily learner-driven activity, which left learners caught in the middle, wanting observation but also wanting to appear independent and efficient. CONCLUSIONS: The cultural values of autonomy in learning and practice and efficiency in health care provision challenge the integration of direct observation into clinical training. Medical learners are often expected to ask for observation, but such requests are socially and culturally fraught, and likely to constrain the wider uptake of direct observation.

Do OSCE progress test scores predict performance in a national high-stakes examination?

CONTEXT: Progress tests, in which learners are repeatedly assessed on equivalent content at different times in their training and provided with feedback, would seem to lend themselves well to a competency-based framework, which requires more frequent formative assessments. The objective structured clinical examination (OSCE) progress test is a relatively new form of assessment that is used to assess the progression of clinical skills. The purpose of this study was to establish further evidence for the use of an OSCE progress test by demonstrating an association between scores from this assessment method and those from a national high-stakes examination. METHODS: The results of 8 years' of data from an Internal Medicine Residency OSCE (IM-OSCE) progress test were compared with scores on the Royal College of Physicians and Surgeons of Canada Comprehensive Objective Examination in Internal Medicine (RCPSC IM examination), which is comprised of both a written and performance-based component (n = 180). Correlations between scores in the two examinations were calculated. Logistic regression analyses were performed comparing IM-OSCE progress test scores with an 'elevated risk of failure' on either component of the RCPSC IM examination. RESULTS: Correlations between scores from the IM-OSCE (for PGY-1 residents to PGY-4 residents) and those from the RCPSC IM examination ranged from 0.316 (p = 0.001) to 0.554 (<.001) for the performance-based component and 0.305 (p = 0.002) to 0.516 (p < 0.001) for the written component. Logistic regression models demonstrated that PGY-2 and PGY-4 scores from the IM-OSCE were predictive of an 'elevated risk of failure' on both components of the RCPSC IM examination. CONCLUSIONS: This study provides further evidence for the use of OSCE progress testing by demonstrating a correlation between scores from an OSCE progress test and a national high-stakes examination. Furthermore, there is evidence that OSCE progress test scores are predictive of future performance on a national high-stakes examination.

A contemporary approach to validity arguments: a practical guide to Kane's framework.

CONTEXT: Assessment is central to medical education and the validation of assessments is vital to their use. Earlier validity frameworks suffer from a multiplicity of types of validity or failure to prioritise among sources of validity evidence. Kane's framework addresses both concerns by emphasising key inferences as the assessment progresses from a single observation to a final decision. Evidence evaluating these inferences is planned and presented as a validity argument. OBJECTIVES: We aim to offer a practical introduction to the key concepts of Kane's framework that educators will find accessible and applicable to a wide range of assessment tools and activities. RESULTS: All assessments are ultimately intended to facilitate a defensible decision about the person being assessed. Validation is the process of collecting and interpreting evidence to support that decision. Rigorous validation involves articulating the claims and assumptions associated with the proposed decision (the interpretation/use argument), empirically testing these assumptions, and organising evidence into a coherent validity argument. Kane identifies four inferences in the validity argument: Scoring (translating an observation into one or more scores); Generalisation (using the score[s] as a reflection of performance in a test setting); Extrapolation (using the score[s] as a reflection of real-world performance), and Implications (applying the score[s] to inform a decision or action). Evidence should be collected to support each of these inferences and should focus on the most questionable assumptions in the chain of inference. Key assumptions (and needed evidence) vary depending on the assessment's intended use or associated decision. Kane's framework applies to quantitative and qualitative assessments, and to individual tests and programmes of assessment. CONCLUSIONS: Validation focuses on evaluating the key claims, assumptions and inferences that link assessment scores with their intended interpretations and uses. The Implications and associated decisions are the most important inferences in the validity argument.