Programmatic Assessment of Clinical Reasoning: New Opportunities to Meet an Ongoing Challenge.

Issue: Clinical reasoning is essential to physicians' competence, yet assessment of clinical reasoning remains a significant challenge. Clinical reasoning is a complex, evolving, non-linear, context-driven, and content-specific construct which arguably cannot be assessed at one point in time or with a single method. This has posed challenges for educators for many decades, despite significant development of individual assessment methods. Evidence: Programmatic assessment is a systematic assessment approach that is gaining momentum across health professions education. Programmatic assessment, and in particular assessment for learning, is well-suited to address the challenges with clinical reasoning assessment. Several key principles of programmatic assessment are particularly well-aligned with developing a system to assess clinical reasoning: longitudinality, triangulation, use of a mix of assessment methods, proportionality, implementation of intermediate evaluations/reviews with faculty coaches, use of assessment for feedback, and increase in learners' agency. Repeated exposure and measurement are critical to develop a clinical reasoning assessment narrative, thus the assessment approach should optimally be longitudinal, providing multiple opportunities for growth and development. Triangulation provides a lens to assess the multidimensionality and contextuality of clinical reasoning and that of its different, yet related components, using a mix of different assessment methods. Proportionality ensures the richness of information on which to draw conclusions is commensurate with the stakes of the decision. Coaching facilitates the development of a feedback culture and allows to assess growth over time, while enhancing learners' agency. Implications: A programmatic assessment model of clinical reasoning that is developmentally oriented, optimizes learning though feedback and coaching, uses multiple assessment methods, and provides opportunity for meaningful triangulation of data can help address some of the challenges of clinical reasoning assessment.

What Stops Fairness from Emerging in Assessment? The Forces on a Complex Adaptive System.

Introduction: Workplace-based assessment occurs in authentic, dynamic clinical environments where reproducible, measurement-based assessments can often not be implemented. In these environments, research approaches that respect these multiple dynamic interactions, such as complexity perspectives, are encouraged. Previous research has shown that fairness in assessment is a nonlinear phenomenon that emerges from interactions between its components and behaves like a complex adaptative system. The aim of this study was to understand the external forces on the complex adaptive system which may disrupt fairness from emerging. Methods: We conducted online focus groups with a purposeful sample of nineteen academic leaders in the Netherlands. We used an iterative approach to collection, analysis and coding of the data and interpreted the results using a lens of complexity, focusing on how individual elements of fairness work in concert to create systems with complex behaviour. Results: We identified three themes of forces which can disrupt fairness: forces impairing interactivity, forces impairing adaption and forces impairing embeddedness. Within each of these themes, we identified subthemes: assessor and student forces, tool forces and system forces. Discussion: Consistent with complexity theory, this study suggests there are multiple forces which can hamper the emergence of fairness. Whilst complexity thinking does not reduce the scale of the challenge, viewing forces through this lens provides insight into why and how these forces are disrupting fairness. This allows for more purposeful, meaningful changes to support the use of fair judgement in assessment in dynamic authentic clinical workplaces.

Fairness in Assessment: Identifying a Complex Adaptive System.

Introduction: Assessment design in health professions education is continuously evolving. There is an increasing desire to better embrace human judgement in assessment. Thus, it is essential to understand what makes this judgement fair. This study builds upon existing literature by studying how assessment leaders conceptualise the characteristics of fair judgement. Methods: Sixteen assessment leaders from 15 medical schools in Australia and New Zealand participated in online focus groups. Data collection and analysis occurred concurrently and iteratively. We used the constant comparison method to identify themes and build on an existing conceptual model of fair judgement in assessment. Results: Fairness is a multi-dimensional construct with components at environment, system and individual levels. Components influencing fairness include articulated and agreed learning outcomes relating to the needs of society, a culture which allows for learner support, stakeholder agency and learning (environmental level), collection, interpretation and combination of evidence, procedural strategies (system level) and appropriate individual assessments and assessor expertise and agility (individual level). Discussion: We observed that within the data at fractal, that is an infinite pattern repeating at different scales, could be seen suggesting fair judgement should be considered a complex adaptive system. Within complex adaptive systems, it is primarily the interaction between the entities which influences the outcome it produces, not simply the components themselves. Viewing fairness in assessment through a lens of complexity rather than as a linear, causal model has significant implications for how we design assessment programs and seek to utilise human judgement in assessment.

The importance of developing meaningfulness and manageability for resilience in rural doctors.

INTRODUCTION: Retention of rural doctors is a problem in many countries. A previous study has identified resilience as a factor associated with longer retention. However, this needs a deeper study to understand what local and personal factors are at play. Studies suggest resilience can be developed during training. We propose that a better understanding of factors associated with resilience might assist in training students for rural practice and increase retention. AIM: This study aimed to understand the differences in resilience development between the more and the less resilient rural doctors. A secondary purpose was to identify how to assist this developmental process through health professional education. METHODS: This study employed a mixed-method design and was part of a more extensive study aiming to develop rural doctors' resilience in a low-resource setting. A prior survey assessed rural doctors' resilience levels. This study sampled high-level and low-level resilience participants to be interviewed. A total of 22 rural doctors participated in the individual semi-structured interviews. The interviews were analyzed qualitatively based on Richardson's Resilience Model and the six resilience dimensions looking for factors that explained high or low resilience. RESULTS: Two important themes emerged during the qualitative analysis: 'meaningfulness' and 'manageability.' The different responses of high and low-resilient participants can be explained through cases. CONCLUSIONS: The participants' perceived meaningfulness and manageability of the stressor determine the responses. We suggest that teachers may better construct students' resilience by focussing on assisting them in finding meaning and developing a sense of manageability.

[Fair assessment judgements of specialty registrars in clinical practise]. / Eerlijk beoordelen van aiossen in de klinische praktijk.

Forming judgements about registrars in a workplace-based assessment context is not easy. Of course, the workplace is an important environment for learning, but it's also often very busy and sometimes complex. Assessment judgements can, therefore, not always be standardised and structured. Nevertheless, judgements have to be defensible and fair. In this paper we will discuss the main guidelines to ensure that assessment judgements are defensible and fair. We will approach these from a perspective of assessment for learning - with the intent to use feedback in order to guide learning - and from the perspective of assessment of learning, using judgements to determine whether a registrar is ready to progress to a next phase. Core elements of good judgement are transparency in the form of constructive and actionable feedback in dialogue with the registrar, and procedural fairness in the form of multiple judgements, over time and with clear documentation. This all needs to be supported by a training quality assurance system.

An international study on the implementation of programmatic assessment: Understanding challenges and exploring solutions.

INTRODUCTION: Programmatic assessment is an approach to assessment aimed at optimizing the learning and decision function of assessment. It involves a set of key principles and ground rules that are important for its design and implementation. However, despite its intuitive appeal, its implementation remains a challenge. The purpose of this paper is to gain a better understanding of the factors that affect the implementation process of programmatic assessment and how specific implementation challenges are managed across different programs. METHODS: An explanatory multiple case (collective) approach was used for this study. We identified 6 medical programs that had implemented programmatic assessment with variation regarding health profession disciplines, level of education and geographic location. We conducted interviews with a key faculty member from each of the programs and analyzed the data using inductive thematic analysis. RESULTS: We identified two major factors in managing the challenges and complexity of the implementation process: knowledge brokers and a strategic opportunistic approach. Knowledge brokers were the people who drove and designed the implementation process acting by translating evidence into practice allowing for real-time management of the complex processes of implementation. These knowledge brokers used a 'strategic opportunistic' or agile approach to recognize new opportunities, secure leadership support, adapt to the context and take advantage of the unexpected. Engaging in an overall curriculum reform process was a critical factor for a successful implementation of programmatic assessment. DISCUSSION: The study contributes to the understanding of the intricacies of implementation processes of programmatic assessment across different institutions. Managing opportunities, adaptive planning, awareness of context, were all critical aspects of thinking strategically and opportunistically in the implementation of programmatic assessment. Future research is needed to provide a more in-depth understanding of values and beliefs that underpin the assessment culture of an organization, and how such values may affect implementation.

The pursuit of fairness in assessment: Looking beyond the objective.

Health professions education has undergone significant changes over the last few decades, including the rise of competency-based medical education, a shift to authentic workplace-based assessments, and increased emphasis on programmes of assessment. Despite these changes, there is still a commonly held assumption that objectivity always leads to and is the only way to achieve fairness in assessment. However, there are well-documented limitations to using objectivity as the 'gold standard' to which assessments are judged. Fairness, on the other hand, is a fundamental quality of assessment and a principle that almost no one contests. Taking a step back and changing perspectives to focus on fairness in assessment may help re-set a traditional objective approach and identify an equal role for subjective human judgement in assessment alongside objective methods. This paper explores fairness as a fundamental quality of assessments. This approach legitimises human judgement and shared subjectivity in assessment decisions alongside objective methods. Widening the answer to the question: 'What is fair assessment' to include not only objectivity but also expert human judgement and shared subjectivity can add significant value in ensuring learners are better equipped to be the health professionals required of the 21st century.

Advancing the science of health professions education through a shared understanding of terminology: a content analysis of terms for "faculty".

INTRODUCTION: Health professions educators risk misunderstandings where terms and concepts are not clearly defined, hampering the field's progress. This risk is especially pronounced with ambiguity in describing roles. This study explores the variety of terms used by researchers and educators to describe "faculty", with the aim to facilitate definitional clarity, and create a shared terminology and approach to describing this term. METHODS: The authors analyzed journal article abstracts to identify the specific words and phrases used to describe individuals or groups of people referred to as faculty. To identify abstracts, PubMed articles indexed with the Medical Subject Heading "faculty" published between 2007 and 2017 were retrieved. Authors iteratively extracted data and used content analysis to identify patterns and themes. RESULTS: A total of 5,436 citations were retrieved, of which 3,354 were deemed eligible. Based on a sample of 594 abstracts (17.7%), we found 279 unique terms. The most commonly used terms accounted for approximately one-third of the sample and included faculty or faculty member/s (n = 252; 26.4%); teacher/s (n = 59; 6.2%) and medical educator/s (n = 26; 2.7%) were also well represented. Content analysis highlighted that the different descriptors authors used referred to four role types: healthcare (e.g., doctor, physician), education (e.g., educator, teacher), academia (e.g., professor), and/or relationship to the learner (e.g., mentor). DISCUSSION: Faculty are described using a wide variety of terms, which can be linked to four role descriptions. The authors propose a template for researchers and educators who want to refer to faculty in their papers. This is important to advance the field and increase readers' assessment of transferability.

Even a little sleepiness influences neural activation and clinical reasoning in novices.

BACKGROUND AND AIMS: Sleepiness influences alertness and cognitive functioning and impacts many aspects of medical care, including clinical reasoning. However, dual processing theory suggests that sleepiness will impact clinical reasoning differently in different individual, depending on their level of experience with the given condition. Our aim, therefore, was to examine the association between clinical reasoning, neuroanatomical activation, and sleepiness in senior medical students. METHODS: Our methodology replicated an earlier study but with novices rather than board-certified physicians. Eighteen final-year medical students answered validated multiple-choice questions (MCQs) during an fMRI scan. Each MCQ was projected in three phases: reading, answering, and reflection (modified think aloud). Echo-planar imaging (EPI) scans gave a time series that reflected blood oxygenation level dependent (BOLD) signal in each location (voxel) within the brain. Sleep data were collected via self-report (Epworth Sleepiness Scale) and actigraphy. These data were correlated with answer accuracy using Pearson correlation. RESULTS: Analysis revealed an increased BOLD signal in the right dorsomedial prefrontal cortex (P < .05) during reflection (Phase 3) associated with increased self-reported sleepiness (ESS) immediately before scanning. Covariate analysis also revealed that increased BOLD signal in the right supramarginal gyrus (P < .05) when reflecting (Phase 3) was associated with increased correct answer response time. Both patterns indicate effortful analytic (System 2) reasoning. CONCLUSION: Our findings that novices use System 2 thinking for clinical reasoning and even a little (perceived) sleepiness influences their clinical reasoning ability to suggest that the parameters for safe working may be different for novices (eg, junior doctors) and experienced physicians.

Developing Personal Resilience Questionnaire for rural doctors: an indigenous approach study in Indonesia.

BACKGROUND: Resilience is recognized as a critical component of well-being and is an essential factor in coping with stress. There are issues of using a standardized resilience scale developed for one cultural population to be used in the different cultural populations. This study aimed to create a specific measurement scale for measuring doctors' resilience levels in the rural Indonesian context. METHOD: A total of 527 rural doctors and health professional educators joined this study (37 and 490 participants in the pilot studies and the survey, respectively). An indigenous psychological approach was implemented in linguistic and cultural adaptation and validation of an existing instrument into the local Indonesian rural health context. A combined method of back-translation, committee approach, communication with the original author, and exploratory qualitative study in the local context was conducted. The indigenous psychological approach was implemented in exploring the local context and writing additional local items. RESULT: The final questionnaire consisted of six dimensions and 30 items with good internal consistency (Cronbach's α ranged 0.809-0.960 for each dimension). Ten locally developed items were added to the final questionnaire as a result of the indigenous psychological approach. CONCLUSION: An indigenous psychological approach may enrich the linguistic and cultural adaptation and validation process of an existing scale.

Exploring complexities in the reform of assessment practice: a critical realist perspective.

Although the principles behind assessment for and as learning are well-established, there can be a struggle when reforming traditional assessment of learning to a program which encompasses assessment for and as learning. When introducing and reporting reforms, tensions in faculty may arise because of differing beliefs about the relationship between assessment and learning and the rules for the validity of assessments. Traditional systems of assessment of learning privilege objective, structured quantification of learners' performances, and are done to the students. Newer systems of assessment promote assessment for learning, emphasise subjectivity, collate data from multiple sources, emphasise narrative-rich feedback to promote learner agency, and are done with the students. This contrast has implications for implementation and evaluative research. Research of assessment which is done to students typically asks, "what works", whereas assessment that is done with the students focuses on more complex questions such as "what works, for whom, in which context, and why?" We applied such a critical realist perspective drawing on the interplay between structure and agency, and a systems approach to explore what theory says about introducing programmatic assessment in the context of pre-existing traditional approaches. Using a reflective technique, the internal conversation, we developed four factors that can assist educators considering major change to assessment practice in their own contexts. These include enabling positive learner agency and engagement; establishing argument-based validity frameworks; designing purposeful and eclectic evidence-based assessment tasks; and developing a shared narrative that promotes reflexivity in appreciating the complex relationships between assessment and learning.

How culture affects validity: understanding Japanese residents' sense-making of evaluating clinical teachers.

OBJECTIVES: Traditionally, evaluation is considered a measurement process that can be performed independently of the cultural context. However, more recently the importance of considering raters' sense-making, that is, the process by which raters assign meaning to their collective experiences, is being recognised. Thus far, the majority of the discussion on this topic has originated from Western perspectives. Little is known about the potential influence of an Asian culture on raters' sense-making. This study explored residents' sense-making associated with evaluating their clinical teachers within an Asian setting to better understand contextual dependency of validity. DESIGN: A qualitative study using constructivist grounded theory. SETTING: The Japanese Ministry of Health, Labour and Welfare has implemented a system to monitor the quality of clinical teaching within its 2-year postgraduate training programme. An evaluation instrument was developed specifically for the Japanese setting through which residents can evaluate their clinical teachers. PARTICIPANTS: 30 residents from 10 Japanese teaching hospitals with experience in evaluating their clinical teachers were sampled purposively and theoretically. METHODS: We conducted in-depth semistructured individual interviews. Sensitising concepts derived from Confucianism and principles of response process informed open, axial and selective coding. RESULTS: Two themes and four subthemes were constructed. Japanese residents emphasised the awareness of their relationship with their clinical teachers (1). This awareness was fuelled by their sense of hierarchy (1a) and being part of the collective society (1b). Residents described how the meaning of evaluation (2) was coloured by their perceived role as senior (2a) and their experienced responsibility for future generations (2b). CONCLUSIONS: Japanese residents' sense-making while evaluating their clinical teachers appears to be situated and affected by Japanese cultural values. These findings contribute to a better understanding of a culture's influence on residents' sense-making of evaluation instruments and the validity argument of evaluation.

Making it fair: Learners' and assessors' perspectives of the attributes of fair judgement.

INTRODUCTION: Optimising the use of subjective human judgement in assessment requires understanding what makes judgement fair. Whilst fairness cannot be simplistically defined, the underpinnings of fair judgement within the literature have been previously combined to create a theoretically-constructed conceptual model. However understanding assessors' and learners' perceptions of what is fair human judgement is also necessary. The aim of this study is to explore assessors' and learners' perceptions of fair human judgement, and to compare these to the conceptual model. METHODS: A thematic analysis approach was used. A purposive sample of twelve assessors and eight post-graduate trainees undertook semi-structured interviews using vignettes. Themes were identified using the process of constant comparison. Collection, analysis and coding of the data occurred simultaneously in an iterative manner until saturation was reached. RESULTS: This study supported the literature-derived conceptual model suggesting fairness is a multi-dimensional construct with components at individual, system and environmental levels. At an individual level, contextual, longitudinally-collected evidence, which is supported by narrative, and falls within ill-defined boundaries is essential for fair judgement. Assessor agility and expertise are needed to interpret and interrogate evidence, identify boundaries and provide narrative feedback to allow for improvement. At a system level, factors such as multiple opportunities to demonstrate competence and improvement, multiple assessors to allow for different perspectives to be triangulated, and documentation are needed for fair judgement. These system features can be optimized through procedural fairness. Finally, appropriate learning and working environments which considers patient needs and learners personal circumstances are needed for fair judgments. DISCUSSION: This study builds on the theory-derived conceptual model demonstrating the components of fair judgement can be explicitly articulated whilst embracing the complexity and contextual nature of health-professions assessment. Thus it provides a narrative to support dialogue between learner, assessor and institutions about ensuring fair judgements in assessment.

Knowledge to action: a scoping review of approaches to educate primary care providers in the identification and management of routine sleep disorders.

STUDY OBJECTIVES: The referral burden on health care systems for routine sleep disorders could be alleviated by educating primary care providers (PCPs) to diagnose and manage patients with sleep health issues. This requires effective professional education strategies and resources. This scoping review examined the literature on existing approaches to educate PCPs in sleep health management. METHODS: A comprehensive literature search was conducted across 8 databases to identify citations describing the education of PCPs in diagnosing and managing sleep disorders, specifically insomnia and sleep apnea. A conceptual framework, developed from the knowledge-to-action cycle, was used to analyze citations from a knowledge translation perspective. RESULTS: Searches identified 616 unique citations and after selection criteria were applied, 22 reports were included. Reports spanning 38 years were analyzed using components of the knowledge-to-action cycle to understand how educational interventions were designed, developed, implemented, and evaluated. Interventions involved didactic (32%), active (18%), and blended (41%) approaches, using face-to-face (27%), technology-mediated (45%), and multimodal (5%) delivery. Educational effectiveness was assessed in 73% of reports, most commonly using a pre/post questionnaire (41%). CONCLUSIONS: While this scoping review has utility in describing existing educational interventions to upskill PCPs to diagnose and manage sleep disorders, the findings suggest that interventions are often developed without explicitly considering the evidence of best educational practice. Future interventional designs may achieve greater sustained effectiveness by considering characteristics of the target audience, the pedagogical approaches best suited to its needs, and any environmental drivers and barriers that might impede the translation of evidence into practice. CITATION: King S, Damarell R, Schuwirth L, Vakulin A, Chai-Coetzer CL, McEvoy RD. Knowledge to action: a scoping review of approaches to educate primary care providers in the identification and management of routine sleep disorders. J Clin Sleep Med. 2021;17(11):2307-2324.

Identifying the at-risk General Practice trainee: a retrospective cohort meta-analysis of General Practice registrar flagging.

A central principle of programmatic assessment is that the final decision is not a surprise to the learner. To achieve this, assessments must demonstrate predictive and consequential validity, however, to date, research has only focussed on the former. The present study attempts to address this gap by examining the predictive and consequential validity of flagging systems used by Australian General Practice regional training organisations (RTOs) in relation to Fellowship examinations. Informed by unstructured interviews with Senior Medical Educators to understand the flagging system of each RTO, meta-analyses of routinely-collected flagging data were used to examine the predictive validity of flagging at various points in training and exam performance. Additionally, flagging system features identified from the interviews were used to inform exploratory subgroup analyses and meta-regressions to further assess the predictive and consequential validity of these systems. Registrars flagged near the end of their training were two to four times more likely to fail Fellowship exams than their non-flagged counterparts. Regarding flagging system features, having graded (i.e. ordinal) flagging systems was associated with higher accuracy, whilst involving the assigned medical educator in remediation and initiating a formal diagnostic procedure following a flag improved registrars' chances of passing exams. These results demonstrate both predictive and consequential validity of flagging systems. We argue that flagging is most effective when initiated early in training in conjunction with mechanisms to maximise diagnostic accuracy and the quality of remediation programs.