The impact of systematically repairing multiple choice questions with low discrimination on assessment reliability: an interrupted time series analysis.

At our centre, we introduced a continuous quality improvement (CQI) initiative during academic year 2018-19 targeting for repair multiple choice question (MCQ) items with discrimination index (D) < 0.1. The purpose of this study was to assess the impact of this initiative on reliability/internal consistency of our assessments. Our participants were medical students during academic years 2015-16 to 2020-21 and our data were summative MCQ assessments during this time. Since the goal was to systematically review and improve summative assessments in our undergraduate program on an ongoing basis, we used interrupted time series analysis to assess the impact on reliability. Between 2015-16 and 2017-18 there was a significant negative trend in the mean alpha coefficient for MCQ exams (regression coefficient -0.027 [-0.008, -0.047], p = 0.024). In the academic year following the introduction of our initiative (2018-19) there was a significant increase in the mean alpha coefficient (regression coefficient 0.113 [0.063, 0.163], p = 0.010) which was then followed by a significant positive post-intervention trend (regression coefficient 0.056 [0.037, 0.075], p = 0.006). In conclusion, our CQI intervention resulted in an immediate and progressive improvement reliability of our MCQ assessments.

Dans notre centre, nous avons introduit une initiative d'amélioration continue de la qualité (ACQ) au cours de l'année académique 2018-19 ciblant la correction des questions à choix multiples (QCM) dont l'indice de discrimination (D) est < 0,1. Le but de cette étude était d'évaluer l'impact de cette initiative sur la fiabilité/cohérence interne de nos évaluations. Nos participants étaient des étudiants en médecine au cours des années académiques 2015-16 à 2020-21 et nos données provenaient d'évaluations sommatives par QCM au cours de cette période. Comme l'objectif était de revoir et d'améliorer systématiquement les évaluations sommatives dans notre programme prégradué sur une base continue, nous avons utilisé une analyse basée sur des séries chronologies interrompues pour évaluer l'impact sur la fiabilité. Entre 2015-16 et 2017-18, il y a eu une tendance négative significative dans le coefficient alpha moyen pour les examens utilisant des QCM (coefficient de régression -0,027 [-0,008, -0,047], p = 0,024). Au cours de l'année académique suivant l'introduction de notre initiative (2018-19), il y a eu une augmentation significative du coefficient alpha moyen (coefficient de régression 0,113 [0,063, 0,163], p = 0,010) qui a été suivie d'une tendance positive significative après l'intervention (coefficient de régression 0,056 [0,037, 0,075], p = 0,006). En conclusion, notre intervention d'ACQ a entraîné une amélioration immédiate et progressive de la fiabilité de nos évaluations par QCM.

Personal career decisions during medical training are not complicated, they are complex.

BACKGROUND: For medical training to be deemed successful, in addition to gaining the skills required to make appropriate clinical decisions, trainees must learn how to make good personal decisions. These decisions may affect satisfaction with career choice, work-life balance, and their ability to maintain/improve clinical performance over time-outcomes that can impact future wellness. Here, the authors introduce a decision-making framework with the goal of improving our understanding of personal decisions. METHODS: Stemming from the business world, the Cynefin framework describes five decision-making domains: clear, complicated, complex, chaotic, and confusion, and a key inference of this framework is that decision-making can be improved by first identifying the decision-making domain. Personal decisions are largely complex-so applying linear decision-making strategies is unlikely to help in this domain. RESULTS: The available data suggest that the outcomes of personal decisions are suboptimal, and the authors propose three mechanisms to explain these findings: (1) Complex decision is susceptible to attribute substitution where we subconsciously trade these decisions for easier decisions; (2) predictions are prone to cognitive biases, such as assuming our situation will remain constant (linear projection fallacy), believing that accomplishing a goal will deliver lasting happiness (arrival bias), or overestimating benefits and underestimating costs of future tasks (planning fallacy); and (3) complex decisions have an inherently higher failure rate than complicated decisions because they are the result of an ongoing, dynamic person-by-situation interaction and, as such, have more time to fail and more ways to do so. DISCUSSION: Based upon their view that personal decisions are complex, the authors propose strategies to improve satisfaction with personal decisions, including increasing awareness of biases that may impact personal decisions. Recognising that the outcome of personal decisions can change over time, they also suggest additional interventions to manage these decisions, such as different forms of mentoring.

A case of hypercalcemia from Pneumocystis jirovecii in an immunosuppressed non-HIV patient.

BACKGROUND: The prevalence of non-HIV related Pneumocystis jirovecii pneumonia (PJP) is increasing with use of immunosuppressive therapies. There are case reports of solid organ transplant recipients on immunosuppressive therapy presenting with mild hypercalcemia, leading to a diagnosis of PJP. Recent studies have shown efficacy of PJP prophylaxis for patients treated with rituximab with a favourable adverse effect profile. CASE PRESENTATION: A 78-year-old male with a history of PR3-ANCA vasculitis, chronic kidney disease and heart failure with reduced ejection fraction presented to our tertiary care hospital with a two-week history of confusion and non-productive cough. Background immunosuppression with rituximab was completed every six months. The patient was found to have hypercalcemia and new infiltrates and ground glass opacities on cross-sectional imaging. Bronchoscopy was performed that was positive for Pneumocystis jirovecii. He was treated with 21 days of trimethoprim-sulfamethoxazole and prednisone with resolution of symptoms and hypercalcemia. CONCLUSIONS: Herein, we present a novel case of PJP in a non-transplant recipient preceded by hypercalcemia. Our case demonstrates the importance for a high suspicion for PJP in chronically immunosuppressed patients on rituximab presenting with PTH-independent hypercalcemia.

Point-of-Care Lung Ultrasound in Emergency Medicine: A Scoping Review With an Interactive Database.

BACKGROUND: This scoping review was conducted to provide an overview of the evidence of point-of-care lung ultrasound (LUS) in emergency medicine. By emphasizing clinical topics, time trends, study designs, and the scope of the primary outcomes, a map is provided for physicians and researchers to guide their future initiatives. RESEARCH QUESTION: Which study designs and primary outcomes are reported in published studies of LUS in emergency medicine? STUDY DESIGN AND METHODS: We performed a systematic search in the PubMed/MEDLINE, Embase, Web of Science, Scopus, and Cochrane Library databases for LUS studies published prior to May 13, 2023. Study characteristics were synthesized quantitatively. The primary outcomes in all papers were categorized into the hierarchical Fryback and Thornbury levels. RESULTS: A total of 4,076 papers were screened and, following selection and handsearching, 406 papers were included. The number of publications doubled from January 2020 to May 2023 (204 to 406 papers). The study designs were primarily observational (n = 375 [92%]), followed by randomized (n = 18 [4%]) and case series (n = 13 [3%]). The primary outcome measure concerned diagnostic accuracy in 319 papers (79%), diagnostic thinking in 32 (8%), therapeutic changes in 4 (1%), and patient outcomes in 14 (3%). No increase in the proportions of randomized controlled trials or the scope of primary outcome measures was observed with time. A freely available interactive database was created to enable readers to search for any given interest (https://public.tableau.com/app/profile/blinded/viz/LUSinEM_240216/INFO). INTERPRETATION: Observational diagnostic studies have been produced in abundance, leaving a paucity of research exploring clinical utility. Notably, research exploring whether LUS causes changes to clinical decisions is imperative prior to any further research being made into patient benefits.

Why we should view the decision of medical trainees to cheat as the product of a person-by-situation interaction.

BACKGROUND: Cheating during medical training is a delicate subject matter with varying opinions on the prevalence, causes and gravity of cheating during training. PROPOSED FRAMEWORK: In this article, the authors suggest that the decision to cheat is best viewed as the product of a person-by-situation interaction rather than indicating inherent dishonesty and/or extrinsic motivation in those who participate in cheating. This framework can explain why individuals who would typically default to honesty may participate in cheating if there is perceived justification for cheating and where situational variables, such as ease of cheating, rewards for cheating and perceived risk associated with cheating, make the decision to cheat appear rational. DISCUSSION: They discuss why the impression that there is a culture of cheating can provide perceived justification for medical trainees to cheat if they have the opportunity. They then describe how aspects of medical training and assessment may enable or hinder cheating by trainees. Consistent with the person-by-situation interaction framework, they contend that our response to cheating should include interventions directed at both the person who cheated and situational variables that enabled cheating. Recognising that some forms of cheating may be widespread, difficult to detect and contentious (such as the creation and use of exam reconstructs), their proposal for dealing with suspected and pervasive cheating is to identify and target enabling variables such that the decision to cheat becomes less rational. Their hope is that in so doing, we can gradually nudge trainees and the culture of medical training towards honesty.

The Choice! The challenges of trying to improve medical students' satisfaction with their specialty choices.

The authors describe the residency match as a two-step process. The first step, the Choice, is where students use a combination of intuitive and analytic information processing to select the specialty that they believe will provide fulfilment and work-life balance over their entire career. The second step, the Match, uses a "deferred-acceptance" algorithm to optimize pairing of students and their specialty choices. Despite being the rate-limiting step, in the minds of students and other stakeholders, the outcomes of the Choice have typically been eclipsed by the outcomes of the Match. A recently published study found that during their second year of residency training, one in 14 physicians reported specialty choice regret, which associates with symptoms of burnout in residents. While the obvious solution is to design interventions that improve the specialty choices of students, this approach faces significant challenges, including the fact that: 1) satisfaction with specialty choice is a difficult-to-define construct; 2) specialty choice regret may be misattributed to a poor choice; and 3) choosing is a more complicated process than matching. The authors end by suggesting that if we hope to improve satisfaction with specialty choice then we should begin by defining this, deciding when to assess it, and then creating assessment tools for which there is validity evidence and that can identify the underlying causes of specialty choice regret.

Les auteurs décrivent le jumelage des résidents comme un processus en deux étapes. La première étape, le Choix, est celle où les étudiants utilisent une combinaison de traitement intuitif et analytique de l'information pour sélectionner la spécialité qui, selon eux, leur apportera l'épanouissement et l'équilibre entre leur vie professionnelle et leur vie privée tout au long de leur carrière. La deuxième étape, le Match, utilise un algorithme « d'acceptation différée ¼ pour optimiser le jumelage des étudiants et de leurs choix de spécialité. Bien qu'ils soient l'étape limitante du processus, selon les étudiants et d'autres parties prenantes, les résultats du Choix sont généralement éclipsés par ceux du jumelage. Une étude récemment publiée a révélé que, durant leur deuxième année de résidence, un médecin sur quatorze regrette d'avoir choisi une spécialité, ce qui est associé à des symptômes d'épuisement professionnel chez les résidents. Bien que la solution évidente soit de développer des interventions qui améliorent les choix de spécialité des étudiants, cette approche pose des défis importants, notamment le fait que : 1) la satisfaction à l'égard du choix de la spécialité est un concept difficile à définir ; 2) le regret du choix de la spécialité peut être attribué à tort à un mauvais choix ; et 3) le choix est un processus plus complexe que le jumelage. Les auteurs concluent en suggérant que si nous espérons améliorer la satisfaction à l'égard du choix de la spécialité, nous devrions commencer par définir ce concept, décider quand l'évaluer, puis créer des outils d'évaluation pour lesquels il existe des preuves de validité et qui peuvent identifier les causes sous-jacentes des regrets à l'égard du choix de la spécialité.

Lung ultrasound: A comparison of image interpretation accuracy between curvilinear and phased array transducers.

Introduction: Both curvilinear and phased array transducers are commonly used to perform lung ultrasound (LUS). This study seeks to compare LUS interpretation accuracy of images obtained using a curvilinear transducer with those obtained using a phased array transducer. Methods: We invited 166 internists and trainees to interpret 16 LUS images/cineloops of eight patients in an online survey: eight curvilinear and eight phased array, performed on the same lung location. Images depicted normal lung, pneumothorax, pleural irregularities, consolidation/hepatisation, pleural effusions and B-lines. Primary outcome for each participant is the difference in image interpretation accuracy scores between the two transducers. Results: A total of 112 (67%) participants completed the survey. The mean paired accuracy score difference between the curvilinear and phased array images was 3.0% (95% CI: 0.6 to 5.4%, P = 0.015). For novices, scores were higher on curvilinear images (mean difference: 5.4%, 95% CI: 0.9 to 9.9%, P = 0.020). For non-novices, there were no differences between the two transducers (mean difference: 1.4%, 95% CI: -1.1 to 3.9%, P = 0.263). For pleural-based findings, the mean of the paired differences between transducers was higher in the novice group (estimated mean difference-in-differences: 9.5%, 95% CI: 0.6 to 18.4%; P = 0.036). No difference in mean accuracies was noted between novices and non-novices for non-pleural-based pathologies (estimated mean difference-in-differences: 0.6%, 95% CI to 5.4-6.6%; P = 0.837). Conclusions: Lung ultrasound images obtained using the curvilinear transducer are associated with higher interpretation accuracy than the phased array transducer. This is especially true for novices interpreting pleural-based pathologies.

The association of attentional foci and image interpretation accuracy in novices interpreting lung ultrasound images: an eye-tracking study.

It is unclear, where learners focus their attention when interpreting point-of-care ultrasound (POCUS) images. This study seeks to determine the relationship between attentional foci metrics with lung ultrasound (LUS) interpretation accuracy in novice medical learners. A convenience sample of 14 medical residents with minimal LUS training viewed 8 LUS cineloops, with their eye-tracking patterns recorded. Areas of interest (AOI) for each cineloop were mapped independently by two experts, and externally validated by a third expert. Primary outcome of interest was image interpretation accuracy, presented as a percentage. Eye tracking captured 10 of 14 participants (71%) who completed the study. Participants spent a mean total of 8 min 44 s ± standard deviation (SD) 3 min 8 s on the cineloops, with 1 min 14 s ± SD 34 s spent fixated in the AOI. Mean accuracy score was 54.0% ± SD 16.8%. In regression analyses, fixation duration within AOI was positively associated with accuracy [beta-coefficients 28.9 standardized error (SE) 6.42, P = 0.002). Total time spent viewing the videos was also significantly associated with accuracy (beta-coefficient 5.08, SE 0.59, P < 0.0001). For each additional minute spent fixating within the AOI, accuracy scores increased by 28.9%. For each additional minute spent viewing the video, accuracy scores increased only by 5.1%. Interpretation accuracy is strongly associated with time spent fixating within the AOI. Image interpretation training should consider targeting AOIs.

Mitigating diagnostic performance bias in a skin-tone balanced dermatology curriculum.

INTRODUCTION: Individuals with skin of colour (SoC) have delayed diagnosis and poorer outcomes when presenting with some dermatologic conditions when compared to individuals with light skin (LS). The objective of this study was to determine if diagnostic performance bias can be mitigated by a skin-tone balanced dermatology curriculum. METHODOLOGY: A prospective randomised intervention study occurred over 2 weeks in 2020 at a Canadian medical school. A convenience sample of all first-year medical students (n = 167) was chosen. In week 1, all participants had access to dermatology podcasts and were randomly allocated to receive non-analytic training (NAT; online patient 'cards') on either SoC cases or LS cases. In week 2, all participants received combined training (CT; NAT and analytic training through workshops on how to apply dermatology diagnostic rules for all skin tones). Participating students completed two formative assessments after weeks 1 and 2. RESULTS: Ninety-two students participated in the study. After week 1, both groups had a lower diagnostic performance on SoC (p = 0.0002 and p = 0.002 for students who trained on LS 'cards' and SoC 'cards', respectively). There was a significant decrease in mean skin tone difference in both groups after week 2 (initial training on SoC: 5.8% (SD 12.2) pre, -1.4% (14.7) post, p = 0.007; initial training on LS: 7.8% (15.4) pre, -4.0% (11.8%) post, p = 0.0001). Five students participated in a post-study survey in 2023, and all found the curriculum enhanced their diagnostic skills in SoC. CONCLUSIONS: SoC performance biases of medical students disappeared after CT in a skin tone-balanced dermatology curriculum.

How much is enough? Proposing achievement thresholds for core EPAs of graduating medical students in Canada.

PURPOSE: The transition towards Competency-Based Medical Education at the Cumming School of Medicine was accelerated by the reduced clinical time caused by the COVID-19 pandemic. The purpose of this study was to define a standard protocol for setting Entrustable Professional Activity (EPA) achievement thresholds and examine their feasibility within the clinical clerkship. METHODS: Achievement thresholds for each of the 12 AFMC EPAs for graduating Canadian medical students were set by using sequential rounds of revision by three consecutive groups of stakeholders and evaluation experts. Structured communication was guided by a modified Delphi technique. The feasibility/consequence models of these EPAs were then assessed by tracking their completion by the graduating class of 2021. RESULTS: The threshold-setting process resulted in set EPA achievement levels ranging from 1 to 8 across the 12 AFMC EPAs. Estimates were stable after the first round for 9 of 12 EPAs. 96.27% of EPAs were successfully completed by clerkship students despite the shortened clinical period. Feasibility was predicted by the slowing rate of EPA accumulation overtime during the clerkship. CONCLUSION: The process described led to consensus on EPA achievement thresholds. Successful completion of the assigned thresholds was feasible within the shortened clerkship.[Box: see text].

Can we predict failure in licensure exams from medical students' undergraduate academic performance?

BACKGROUND: In 2015, the Medical Council of Canada increased the minimum pass level for the Medical Council of Canada Qualifying Examination Part I, and students had a higher rate of failure than in previous years. The purpose of this study was to predict students at an increased odds of examination failure to allow for early, targeted interventions. METHODS: We divided our dataset into a derivation cohort and two validation cohorts and used multiple logistic regression to predict licensing examination failure. We then performed receiver operating characteristics and a sensitivity analysis using different cutoffs for explanatory variables to identify the cutoff threshold with the best predictive value at identifying students at increased odds of failure. RESULTS: After multivariate analysis, only pre-clerkship GPA was a significant independent predictor of failure (OR 0.76, 95% CI [0.66, 0.88], p < 0.001). The probability of failure increased steeply when the pre-clerkship GPA fell below 80% and 76% was found to be the most efficient cutoff for predicting failure (OR 9.37, 95% CI [3.08, 38.41]). CONCLUSIONS: Pre-clerkship performance can predict students at increased odds of licensing examination failure. Further studies are needed to explore whether early interventions for at-risk students alter their examination performance.

CONTEXTE: En 2015, le Conseil médical du Canada a resserré les exigences de réussite à l'examen d'aptitude du Conseil médical du Canada, partie I, entraînant un taux d'échec plus élevé que les années précédentes. L'objectif de cette étude était de détecter les étudiants ayant de plus grande probabilité d'échec à l'examen afin de permettre des interventions ciblées en temps utile. MÉTHODES: Nous avons comparé les données d'une cohorte de dérivation et deux cohortes de validation et nous avons utilisé la régression logistique multiple pour prédire l'échec à l'examen d'aptitude. Nous avons ensuite effectué une analyse de la fonction d'efficacité du récepteur et une analyse de sensibilité en utilisant différents seuils pour les variables explicatives afin de déterminer la meilleure valeur prédictive seuil pour cibler une forte possibilité d'échec chez les étudiants. RÉSULTATS: L'analyse multivariée a révélé que seule la moyenne générale des étudiants était un prédicteur indépendant significatif de l'échec (OR 0.76, 95 % CI [0.66, 0.88], p < 0.001). La probabilité d'échec augmentait fortement lorsque l'indice de moyenne générale tombait en dessous de 80 %. Le seuil le plus efficace pour prédire l'échec s'est avéré être 76 % (OR 9,37, 95 % CI [3,08, 38,41]). CONCLUSIONS: Les résultats scolaires des étudiants en médecine constituent un indicateur de risque d'échec à l'examen d'aptitude. Des études supplémentaires sont nécessaires pour vérifier si une intervention précoce auprès des étudiants à risque peut améliorer leurs résultats à l'examen.

Use of Critical Items in Determining Point-of-Care Ultrasound Competence.

We previously developed a workplace-based tool for assessing point of care ultrasound (POCUS) skills and used a modified Delphi technique to identify critical items (those that learners must successfully complete to be considered competent). We performed a standard setting procedure to determine cut scores for the full tool and a focused critical item tool. This study compared ratings by 24 experts on the two checklists versus a global entrustability rating. All experts assessed three videos showing an actor performing a POCUS exam on a patient. The performances were designed to show a range of competences and one included potentially critical errors. Interrater reliability for the critical item tool was higher than for the full tool (intraclass correlation coefficient = 0.84 [95% confidence interval [CI] 0.42-0.99] vs. 0.78 [95% CI 0.25-0.99]). Agreement with global ratings of competence was higher for the critical item tool (κ = 0.71 [95% CI 0.55-0.88] vs 0.48 [95% CI 0.30-0.67]). Although sensitivity was higher for the full tool (85.4% [95% CI 72.2-93.9%] vs. 81.3% [95% CI 67.5-91.1%]), specificity was higher for the critical item tool (70.8% [95% CI 48.9-87.4%] vs. 29.2% [95% CI 12.6-51.1%]). We recommend the use of critical item checklists for the assessment of POCUS competence.

Consensus-Based Expert Development of Critical Items for Direct Observation of Point-of-Care Ultrasound Skills.

BACKGROUND: Point-of-care ultrasound (POCUS) is increasingly used in a number of medical specialties. To support competency-based POCUS education, workplace-based assessments are essential. OBJECTIVE: We developed a consensus-based assessment tool for POCUS skills and determined which items are critical for competence. We then performed standards setting to set cut scores for the tool. METHODS: Using a modified Delphi technique, 25 experts voted on 32 items over 3 rounds between August and December 2016. Consensus was defined as agreement by at least 80% of the experts. Twelve experts then performed 3 rounds of a standards setting procedure in March 2017 to establish cut scores. RESULTS: Experts reached consensus for 31 items to include in the tool. Experts reached consensus that 16 of those items were critically important. A final cut score for the tool was established at 65.2% (SD 17.0%). Cut scores for critical items are significantly higher than those for noncritical items (76.5% ± SD 12.4% versus 53.1% ± SD 12.2%, P < .0001). CONCLUSIONS: We reached consensus on a 31-item workplace-based assessment tool for identifying competence in POCUS. Of those items, 16 were considered critically important. Their importance is further supported by higher cut scores compared with noncritical items.