Programmatic Assessment of Clinical Reasoning: New Opportunities to Meet an Ongoing Challenge.

Issue: Clinical reasoning is essential to physicians' competence, yet assessment of clinical reasoning remains a significant challenge. Clinical reasoning is a complex, evolving, non-linear, context-driven, and content-specific construct which arguably cannot be assessed at one point in time or with a single method. This has posed challenges for educators for many decades, despite significant development of individual assessment methods. Evidence: Programmatic assessment is a systematic assessment approach that is gaining momentum across health professions education. Programmatic assessment, and in particular assessment for learning, is well-suited to address the challenges with clinical reasoning assessment. Several key principles of programmatic assessment are particularly well-aligned with developing a system to assess clinical reasoning: longitudinality, triangulation, use of a mix of assessment methods, proportionality, implementation of intermediate evaluations/reviews with faculty coaches, use of assessment for feedback, and increase in learners' agency. Repeated exposure and measurement are critical to develop a clinical reasoning assessment narrative, thus the assessment approach should optimally be longitudinal, providing multiple opportunities for growth and development. Triangulation provides a lens to assess the multidimensionality and contextuality of clinical reasoning and that of its different, yet related components, using a mix of different assessment methods. Proportionality ensures the richness of information on which to draw conclusions is commensurate with the stakes of the decision. Coaching facilitates the development of a feedback culture and allows to assess growth over time, while enhancing learners' agency. Implications: A programmatic assessment model of clinical reasoning that is developmentally oriented, optimizes learning though feedback and coaching, uses multiple assessment methods, and provides opportunity for meaningful triangulation of data can help address some of the challenges of clinical reasoning assessment.

The importance of developing meaningfulness and manageability for resilience in rural doctors.

INTRODUCTION: Retention of rural doctors is a problem in many countries. A previous study has identified resilience as a factor associated with longer retention. However, this needs a deeper study to understand what local and personal factors are at play. Studies suggest resilience can be developed during training. We propose that a better understanding of factors associated with resilience might assist in training students for rural practice and increase retention. AIM: This study aimed to understand the differences in resilience development between the more and the less resilient rural doctors. A secondary purpose was to identify how to assist this developmental process through health professional education. METHODS: This study employed a mixed-method design and was part of a more extensive study aiming to develop rural doctors' resilience in a low-resource setting. A prior survey assessed rural doctors' resilience levels. This study sampled high-level and low-level resilience participants to be interviewed. A total of 22 rural doctors participated in the individual semi-structured interviews. The interviews were analyzed qualitatively based on Richardson's Resilience Model and the six resilience dimensions looking for factors that explained high or low resilience. RESULTS: Two important themes emerged during the qualitative analysis: 'meaningfulness' and 'manageability.' The different responses of high and low-resilient participants can be explained through cases. CONCLUSIONS: The participants' perceived meaningfulness and manageability of the stressor determine the responses. We suggest that teachers may better construct students' resilience by focussing on assisting them in finding meaning and developing a sense of manageability.

An international study on the implementation of programmatic assessment: Understanding challenges and exploring solutions.

INTRODUCTION: Programmatic assessment is an approach to assessment aimed at optimizing the learning and decision function of assessment. It involves a set of key principles and ground rules that are important for its design and implementation. However, despite its intuitive appeal, its implementation remains a challenge. The purpose of this paper is to gain a better understanding of the factors that affect the implementation process of programmatic assessment and how specific implementation challenges are managed across different programs. METHODS: An explanatory multiple case (collective) approach was used for this study. We identified 6 medical programs that had implemented programmatic assessment with variation regarding health profession disciplines, level of education and geographic location. We conducted interviews with a key faculty member from each of the programs and analyzed the data using inductive thematic analysis. RESULTS: We identified two major factors in managing the challenges and complexity of the implementation process: knowledge brokers and a strategic opportunistic approach. Knowledge brokers were the people who drove and designed the implementation process acting by translating evidence into practice allowing for real-time management of the complex processes of implementation. These knowledge brokers used a 'strategic opportunistic' or agile approach to recognize new opportunities, secure leadership support, adapt to the context and take advantage of the unexpected. Engaging in an overall curriculum reform process was a critical factor for a successful implementation of programmatic assessment. DISCUSSION: The study contributes to the understanding of the intricacies of implementation processes of programmatic assessment across different institutions. Managing opportunities, adaptive planning, awareness of context, were all critical aspects of thinking strategically and opportunistically in the implementation of programmatic assessment. Future research is needed to provide a more in-depth understanding of values and beliefs that underpin the assessment culture of an organization, and how such values may affect implementation.

The pursuit of fairness in assessment: Looking beyond the objective.

Health professions education has undergone significant changes over the last few decades, including the rise of competency-based medical education, a shift to authentic workplace-based assessments, and increased emphasis on programmes of assessment. Despite these changes, there is still a commonly held assumption that objectivity always leads to and is the only way to achieve fairness in assessment. However, there are well-documented limitations to using objectivity as the 'gold standard' to which assessments are judged. Fairness, on the other hand, is a fundamental quality of assessment and a principle that almost no one contests. Taking a step back and changing perspectives to focus on fairness in assessment may help re-set a traditional objective approach and identify an equal role for subjective human judgement in assessment alongside objective methods. This paper explores fairness as a fundamental quality of assessments. This approach legitimises human judgement and shared subjectivity in assessment decisions alongside objective methods. Widening the answer to the question: 'What is fair assessment' to include not only objectivity but also expert human judgement and shared subjectivity can add significant value in ensuring learners are better equipped to be the health professionals required of the 21st century.

Making it fair: Learners' and assessors' perspectives of the attributes of fair judgement.

INTRODUCTION: Optimising the use of subjective human judgement in assessment requires understanding what makes judgement fair. Whilst fairness cannot be simplistically defined, the underpinnings of fair judgement within the literature have been previously combined to create a theoretically-constructed conceptual model. However understanding assessors' and learners' perceptions of what is fair human judgement is also necessary. The aim of this study is to explore assessors' and learners' perceptions of fair human judgement, and to compare these to the conceptual model. METHODS: A thematic analysis approach was used. A purposive sample of twelve assessors and eight post-graduate trainees undertook semi-structured interviews using vignettes. Themes were identified using the process of constant comparison. Collection, analysis and coding of the data occurred simultaneously in an iterative manner until saturation was reached. RESULTS: This study supported the literature-derived conceptual model suggesting fairness is a multi-dimensional construct with components at individual, system and environmental levels. At an individual level, contextual, longitudinally-collected evidence, which is supported by narrative, and falls within ill-defined boundaries is essential for fair judgement. Assessor agility and expertise are needed to interpret and interrogate evidence, identify boundaries and provide narrative feedback to allow for improvement. At a system level, factors such as multiple opportunities to demonstrate competence and improvement, multiple assessors to allow for different perspectives to be triangulated, and documentation are needed for fair judgement. These system features can be optimized through procedural fairness. Finally, appropriate learning and working environments which considers patient needs and learners personal circumstances are needed for fair judgments. DISCUSSION: This study builds on the theory-derived conceptual model demonstrating the components of fair judgement can be explicitly articulated whilst embracing the complexity and contextual nature of health-professions assessment. Thus it provides a narrative to support dialogue between learner, assessor and institutions about ensuring fair judgements in assessment.

Identifying the at-risk General Practice trainee: a retrospective cohort meta-analysis of General Practice registrar flagging.

A central principle of programmatic assessment is that the final decision is not a surprise to the learner. To achieve this, assessments must demonstrate predictive and consequential validity, however, to date, research has only focussed on the former. The present study attempts to address this gap by examining the predictive and consequential validity of flagging systems used by Australian General Practice regional training organisations (RTOs) in relation to Fellowship examinations. Informed by unstructured interviews with Senior Medical Educators to understand the flagging system of each RTO, meta-analyses of routinely-collected flagging data were used to examine the predictive validity of flagging at various points in training and exam performance. Additionally, flagging system features identified from the interviews were used to inform exploratory subgroup analyses and meta-regressions to further assess the predictive and consequential validity of these systems. Registrars flagged near the end of their training were two to four times more likely to fail Fellowship exams than their non-flagged counterparts. Regarding flagging system features, having graded (i.e. ordinal) flagging systems was associated with higher accuracy, whilst involving the assigned medical educator in remediation and initiating a formal diagnostic procedure following a flag improved registrars' chances of passing exams. These results demonstrate both predictive and consequential validity of flagging systems. We argue that flagging is most effective when initiated early in training in conjunction with mechanisms to maximise diagnostic accuracy and the quality of remediation programs.

Exploring complexities in the reform of assessment practice: a critical realist perspective.

Although the principles behind assessment for and as learning are well-established, there can be a struggle when reforming traditional assessment of learning to a program which encompasses assessment for and as learning. When introducing and reporting reforms, tensions in faculty may arise because of differing beliefs about the relationship between assessment and learning and the rules for the validity of assessments. Traditional systems of assessment of learning privilege objective, structured quantification of learners' performances, and are done to the students. Newer systems of assessment promote assessment for learning, emphasise subjectivity, collate data from multiple sources, emphasise narrative-rich feedback to promote learner agency, and are done with the students. This contrast has implications for implementation and evaluative research. Research of assessment which is done to students typically asks, "what works", whereas assessment that is done with the students focuses on more complex questions such as "what works, for whom, in which context, and why?" We applied such a critical realist perspective drawing on the interplay between structure and agency, and a systems approach to explore what theory says about introducing programmatic assessment in the context of pre-existing traditional approaches. Using a reflective technique, the internal conversation, we developed four factors that can assist educators considering major change to assessment practice in their own contexts. These include enabling positive learner agency and engagement; establishing argument-based validity frameworks; designing purposeful and eclectic evidence-based assessment tasks; and developing a shared narrative that promotes reflexivity in appreciating the complex relationships between assessment and learning.

Fairness in human judgement in assessment: a hermeneutic literature review and conceptual framework.

Human judgement is widely used in workplace-based assessment despite criticism that it does not meet standards of objectivity. There is an ongoing push within the literature to better embrace subjective human judgement in assessment not as a 'problem' to be corrected psychometrically but as legitimate perceptions of performance. Taking a step back and changing perspectives to focus on the fundamental underlying value of fairness in assessment may help re-set the traditional objective approach and provide a more relevant way to determine the appropriateness of subjective human judgements. Changing focus to look at what is 'fair' human judgement in assessment, rather than what is 'objective' human judgement in assessment allows for the embracing of many different perspectives, and the legitimising of human judgement in assessment. However, this requires addressing the question: what makes human judgements fair in health professions assessment? This is not a straightforward question with a single unambiguously 'correct' answer. In this hermeneutic literature review we aimed to produce a scholarly knowledge synthesis and understanding of the factors, definitions and key questions associated with fairness in human judgement in assessment and a resulting conceptual framework, with a view to informing ongoing further research. The complex construct of fair human judgement could be conceptualised through values (credibility, fitness for purpose, transparency and defensibility) which are upheld at an individual level by characteristics of fair human judgement (narrative, boundaries, expertise, agility and evidence) and at a systems level by procedures (procedural fairness, documentation, multiple opportunities, multiple assessors, validity evidence) which help translate fairness in human judgement from concepts into practical components.

Prospective Randomized Controlled Trial of Video- Versus Recall-Assisted Reflection in Simulation-Based Teaching on Acquisition and Retention of Airway Skills Among Trainees Intubating Critically Ill Patients.

OBJECTIVES: Conventionally, simulation-based teaching involves reflection on recalled events (recall-assisted reflection). Instead of recall, video-assisted reflection may reduce recall bias and improve skills retention by contributing to visual memory. Here, we test the hypothesis that when compared with recall, video-assisted reflection results in higher acquisition and retention of skills involved in airway management among junior critical care doctors. DESIGN: Randomized control trial. Participants were randomized 1:1 to video-assisted reflection or recall-assisted reflection group. SETTING: University-affiliated tertiary care center. SUBJECTS: Junior critical care doctors. INTERVENTION: Video-assisted reflection. MEASUREMENTS AND MAIN RESULTS: All participants underwent simulation-based teaching of technical and nontechnical airway skills involved in managing a critically ill patient. These skills were assessed before, post-workshop, and in the following fourth week, by two independent blinded assessors using a validated scoring tool. Quality of debrief was assessed using a validated questionnaire. Repeated-measures analysis of variance was used to assess time and group interaction. Forty doctors were randomized. At baseline, the groups had similar airway experience (p = 0.34) and skill scores (p = 0.97). There was a significant interaction between study groups and changes over time for total skill scores (F[2, 37] = 4.06; p = 0.02). Although both the study groups had similar and significant improvement in total skills scores at the postworkshop assessment, the decline in total skills scores at delayed assessment (F[1, 38] = 5.64; p = 0.02) was significantly more in the recall-assisted reflection group when compared with the video-assisted reflection group. This resulted in lower mean skill scores in the recall-assisted reflection group when compared with the video-assisted reflection group in the delayed assessment (89.45 [19.32] vs 110.10 [19.54]; p < 0.01). Better retention was predominantly in the nontechnical skills. The perceived quality of debrief was similar between the two groups. CONCLUSION: When compared with recall, video-assisted reflection resulted in similar improvement in airway skills, but better retention over time.

Workplace-based assessments in postgraduate medical education: A hermeneutic review.

OBJECTIVES: Since their introduction, workplace-based assessments (WBAs) have proliferated throughout postgraduate medical education. Previous reviews have identified mixed findings regarding WBAs' effectiveness, but have not considered the importance of user-tool-context interactions. The present review was conducted to address this gap by generating a thematic overview of factors important to the acceptability, effectiveness and utility of WBAs in postgraduate medical education. METHOD: This review utilised a hermeneutic cycle for analysis of the literature. Four databases were searched to identify articles pertaining to WBAs in postgraduate medical education from the United Kingdom, Canada, Australia, New Zealand, the Netherlands and Scandinavian countries. Over the course of three rounds, 30 published articles were thematically analysed in an iterative fashion to deeply engage with the literature in order to answer three scoping questions concerning acceptability, effectiveness and assessment training. As each round was coded, themes were refined and questions added until saturation was reached. RESULTS: Stakeholders value WBAs for permitting assessment of trainees' performance in an authentic context. Negative perceptions of WBAs stem from misuse due to low assessment literacy, disagreement with definitions and frameworks, and inadequate summative use of WBAs. Effectiveness is influenced by user (eg, engagement and assessment literacy) and tool attributes (eg, definitions and scales), but most fundamentally by user-tool-context interactions, particularly trainee-assessor relationships. Assessors' assessment literacy must be combined with cultural and administrative factors in organisations and the broader medical discipline. CONCLUSIONS: The pivotal determinants of WBAs' effectiveness and utility are the user-tool-context interactions. From the identified themes, we present 12 lessons learned regarding users, tools and contexts to maximise WBA utility, including the separation of formative and summative WBA assessors, use of maximally useful scales, and instituting measures to reduce competitive demands.

A history of assessment in medical education.

The way quality of assessment has been perceived and assured has changed considerably in the recent 5 decades. Originally, assessment was mainly seen as a measurement problem with the aim to tell people apart, the competent from the not competent. Logically, reproducibility or reliability and construct validity were seen as necessary and sufficient for assessment quality and the role of human judgement was minimised. Later, assessment moved back into the authentic workplace with various workplace-based assessment (WBA) methods. Although originally approached from the same measurement framework, WBA and other assessments gradually became assessment processes that included or embraced human judgement but based on good support and assessment expertise. Currently, assessment is treated as a whole system problem in which competence is evaluated from an integrated rather than a reductionist perspective. Current research therefore focuses on how to support and improve human judgement, how to triangulate assessment information meaningfully and how to construct fairness, credibility and defensibility from a systems perspective. But, given the rapid changes in society, education and healthcare, yet another evolution in our thinking about good assessment is likely to lurk around the corner.

Conflict between clinician teachers and their students: the clinician perspective.

The relationship between clinician teachers and their students is of major importance in medical education. However, there is little known about the effects on clinicians when conflict occurs with their students. What do clinicians perceive to be major causes of these conflicts? How do they react when and after conflict occurs? A phenomenological inquiry exploring the lived experience of 12 clinician teachers in medical schools was performed. The clinicians were selected using purposeful sampling and snowballing techniques. The interviews revolved around discussions based on episodes of conflict with medical students that the clinicians considered significant. The analysis and emergent themes were partially constructed around and informed by theories of conflict, and conflict management. A number of themes emerged from this study. Clinicians experienced that significant psychological and behavioural problems of students had a dominant impact on the likelihood and severity of conflict; these conflicts had a significant emotional impact on clinicians; though the responses to conflict varied, "avoidance" was a mechanism commonly used by clinicians and thus the assessment of attitudinal and behavioural professional issues in the workplace was problematic. This study shows how the clinician perspective to challenging student/clinician encounters impacts on the quality of education they are able to provide. We recommend medical schools consider these issues when designing their programs in order to develop and maintain clinician-teacher engagement and participation.

Mapping clinical reasoning literature across the health professions: a scoping review.

BACKGROUND: Clinical reasoning is at the core of health professionals' practice. A mapping of what constitutes clinical reasoning could support the teaching, development, and assessment of clinical reasoning across the health professions. METHODS: We conducted a scoping study to map the literature on clinical reasoning across health professions literature in the context of a larger Best Evidence Medical Education (BEME) review on clinical reasoning assessment. Seven databases were searched using subheadings and terms relating to clinical reasoning, assessment, and Health Professions. Data analysis focused on a comprehensive analysis of bibliometric characteristics and the use of varied terminology to refer to clinical reasoning. RESULTS: Literature identified: 625 papers spanning 47 years (1968-2014), in 155 journals, from 544 first authors, across eighteen Health Professions. Thirty-seven percent of papers used the term clinical reasoning; and 110 other terms referring to the concept of clinical reasoning were identified. Consensus on the categorization of terms was reached for 65 terms across six different categories: reasoning skills, reasoning performance, reasoning process, outcome of reasoning, context of reasoning, and purpose/goal of reasoning. Categories of terminology used differed across Health Professions and publication types. DISCUSSION: Many diverse terms were present and were used differently across literature contexts. These terms likely reflect different operationalisations, or conceptualizations, of clinical reasoning as well as the complex, multi-dimensional nature of this concept. We advise authors to make the intended meaning of 'clinical reasoning' and associated terms in their work explicit in order to facilitate teaching, assessment, and research communication.

Personal resilience and rural doctors retention: a study in Indonesia.

INTRODUCTION: Retaining the health workforce in rural areas is a global problem. Job dissatisfaction or work-related distress are among the factors that drive doctors to leave rural places. Resilience has been recognised as a key component of wellbeing and is associated with better satisfaction with life. Building personal resilience has the benefits of lowering vulnerability to work-related adversity. This study examined the association between rural doctors' personal resilience and the duration of rural practice. METHODS: This cross-sectional study was set in a rural province in Indonesia. A total sampling procedure was implemented. A total of 528 participants responded to an online survey. The survey tool measured six dimensions of a resilience profile (determination, endurance, adaptability, recuperability, comfort zone and life calling) and collected personal data such as date of birth, practice location and duration of rural practice experience. These participants were classified into four groups: intern, general (GP) with 10 years experience. The data were analysed quantitatively using Oneway analysis of variance (ANOVA). RESULTS: Doctors with longer durations of rural experience showed higher resilience levels in four of the dimensions of personal resilience: endurance, adaptability, recuperabilit­y and comfort zone. Among those four dimensions, endurance and comfort zone showed significant differences between groups with >10 years of difference in rural experience (p<0.05). The other two dimensions, determination and life calling, showed fluctuations across groups with different rural durations. CONCLUSION: This study provides a preliminary result for understanding the relationship between personal resilience and rural doctor retention. It suggests that resilience is partly associated with rural doctor retention. Further studies are needed to examine the causal relationship between resilience and retention.

Entrustable professional activities for workplace assessment of general practice trainees.

OBJECTIVE: To assess whether entrustment levels for junior trainees with respect to entrustable professional activities (EPAs) increase over time; whether entrustment levels for senior trainees are higher than for junior trainees; and whether self-assessment of entrustment levels by senior trainees more closely matches supervisor assessment than self-assessment by junior trainees. DESIGN, SETTING, PARTICIPANTS: Observational study of 130 junior and 153 senior community-based general practice trainees in South Australia, 2017. MAIN OUTCOME MEASURES: Differences in entrustment levels between junior and senior trainees; change in entrustment levels for junior trainees over 9 months; concordance of supervisor and trainee assessment of entrustment level over 9 months. RESULTS: Senior trainees were 2.1 (95% CI, 1.66-2.58) to 3.7 times (95% CI, 2.60-5.28) as likely as junior trainees to be entrusted with performing clinical EPAs without supervision. The proportion of EPAs with which junior trainees were entrusted to perform unsupervised increased from 26% at 3 months to 35% at 6 months (rate ratio [RR], 1.37; 95% CI; 1.15-1.63), to 50% at 9 months (RR, 1.92; 95% CI, 1.64-2.26), and 69% at 12 months (RR, 2.68; 95% CI; 2.32-3.12). At 3 months, the mean differences in entrustment ratings between supervisors and trainees was 5.5 points (SD, 6.6 points) for junior trainees and 2.93 points (SD, 2.8 points) for senior trainees (P < 0.001). CONCLUSIONS: EPAs are valid assessment tools in a workplace-based training environment.

Assessment in the context of problem-based learning.

Arguably, constructive alignment has been the major challenge for assessment in the context of problem-based learning (PBL). PBL focuses on promoting abilities such as clinical reasoning, team skills and metacognition. PBL also aims to foster self-directed learning and deep learning as opposed to rote learning. This has incentivized researchers in assessment to find possible solutions. Originally, these solutions were sought in developing the right instruments to measure these PBL-related skills. The search for these instruments has been accelerated by the emergence of competency-based education. With competency-based education assessment moved away from purely standardized testing, relying more heavily on professional judgment of complex skills. Valuable lessons have been learned that are directly relevant for assessment in PBL. Later, solutions were sought in the development of new assessment strategies, initially again with individual instruments such as progress testing, but later through a more holistic approach to the assessment program as a whole. Programmatic assessment is such an integral approach to assessment. It focuses on optimizing learning through assessment, while at the same gathering rich information that can be used for rigorous decision-making about learner progression. Programmatic assessment comes very close to achieving the desired constructive alignment with PBL, but its wide adoption-just like PBL-will take many years ahead of us.

Interventions to improve diagnostic decision making: A systematic review and meta-analysis on reflective strategies.

Aims: To identify, appraise and describe studies of cognitive interventions to improve diagnostic decision making (DDM) amongst medical professionals, assess their effectiveness and identify methodological limitations in existing studies. Methods: We systematically searched for studies (publication date 2000-2016) in multiple databases including Cochrane Controlled Trials, EMBASE, ERIC, Medline, PubMed and PsycINFO, and used additional strategies such as hand searching and snowballing. Included studies evaluated cognitive interventions to enhance DDM amongst medical professionals, using defined outcomes such as diagnostic accuracy. A meta-analysis assessed the impact of "reflection". Results: Forty-four studies out of 10,114 screened citations, involving 4380 medical professionals, were included. Studies evaluated reasoning workshops/curricula, de-biasing workshops, checklists, reflection, feedback, and instructions to induce analytical thinking. Guided reflection was demonstrated to improve DDM [effect size 0.38(95%CI 0.23-0.52), p < 0.001]. Immediate feedback and modeling reflection using contrasting examples also appeared to improve diagnostic accuracy, however underlying methodological issues prevented a quantitative assessment of any strategies other than reflection. Conclusions: Educational interventions incorporating practising deliberate reflection on a formulated diagnosis, modeled reflection on contrasting examples and immediate feedback are promising strategies for improving DDM. The effectiveness of other strategies is unknown, with more methodological refinements required in future research.

The terminology of clinical reasoning in health professions education: Implications and considerations.

Introduction: Clinical reasoning is considered to be at the core of health practice. Here, we report on the diversity and inferred meanings of the terms used to refer to clinical reasoning and consider implications for teaching and assessment. Methods: In the context of a Best Evidence Medical Education (BEME) review of 625 papers drawn from 18 health professions, we identified 110 terms for clinical reasoning. We focus on iterative categorization of these terms across three phases of coding and considerations for how terminology influences educational practices. Results: Following iterative coding with 5 team members, consensus was possible for 74, majority coding was possible for 16, and full team disagreement existed for 20 terms. Categories of terms included: purpose/goal of reasoning, outcome of reasoning, reasoning performance, reasoning processes, reasoning skills, and context of reasoning. Discussion: Findings suggest that terms used in reference to clinical reasoning are non-synonymous, not uniformly understood, and the level of agreement differed across terms. If the language we use to describe, to teach, or to assess clinical reasoning is not similarly understood across clinical teachers, program directors, and learners, this could lead to confusion regarding what the educational or assessment targets are for "clinical reasoning."