The impact of systematically repairing multiple choice questions with low discrimination on assessment reliability: an interrupted time series analysis.

At our centre, we introduced a continuous quality improvement (CQI) initiative during academic year 2018-19 targeting for repair multiple choice question (MCQ) items with discrimination index (D) < 0.1. The purpose of this study was to assess the impact of this initiative on reliability/internal consistency of our assessments. Our participants were medical students during academic years 2015-16 to 2020-21 and our data were summative MCQ assessments during this time. Since the goal was to systematically review and improve summative assessments in our undergraduate program on an ongoing basis, we used interrupted time series analysis to assess the impact on reliability. Between 2015-16 and 2017-18 there was a significant negative trend in the mean alpha coefficient for MCQ exams (regression coefficient -0.027 [-0.008, -0.047], p = 0.024). In the academic year following the introduction of our initiative (2018-19) there was a significant increase in the mean alpha coefficient (regression coefficient 0.113 [0.063, 0.163], p = 0.010) which was then followed by a significant positive post-intervention trend (regression coefficient 0.056 [0.037, 0.075], p = 0.006). In conclusion, our CQI intervention resulted in an immediate and progressive improvement reliability of our MCQ assessments.

Dans notre centre, nous avons introduit une initiative d'amélioration continue de la qualité (ACQ) au cours de l'année académique 2018-19 ciblant la correction des questions à choix multiples (QCM) dont l'indice de discrimination (D) est < 0,1. Le but de cette étude était d'évaluer l'impact de cette initiative sur la fiabilité/cohérence interne de nos évaluations. Nos participants étaient des étudiants en médecine au cours des années académiques 2015-16 à 2020-21 et nos données provenaient d'évaluations sommatives par QCM au cours de cette période. Comme l'objectif était de revoir et d'améliorer systématiquement les évaluations sommatives dans notre programme prégradué sur une base continue, nous avons utilisé une analyse basée sur des séries chronologies interrompues pour évaluer l'impact sur la fiabilité. Entre 2015-16 et 2017-18, il y a eu une tendance négative significative dans le coefficient alpha moyen pour les examens utilisant des QCM (coefficient de régression -0,027 [-0,008, -0,047], p = 0,024). Au cours de l'année académique suivant l'introduction de notre initiative (2018-19), il y a eu une augmentation significative du coefficient alpha moyen (coefficient de régression 0,113 [0,063, 0,163], p = 0,010) qui a été suivie d'une tendance positive significative après l'intervention (coefficient de régression 0,056 [0,037, 0,075], p = 0,006). En conclusion, notre intervention d'ACQ a entraîné une amélioration immédiate et progressive de la fiabilité de nos évaluations par QCM.

How much is enough? Proposing achievement thresholds for core EPAs of graduating medical students in Canada.

PURPOSE: The transition towards Competency-Based Medical Education at the Cumming School of Medicine was accelerated by the reduced clinical time caused by the COVID-19 pandemic. The purpose of this study was to define a standard protocol for setting Entrustable Professional Activity (EPA) achievement thresholds and examine their feasibility within the clinical clerkship. METHODS: Achievement thresholds for each of the 12 AFMC EPAs for graduating Canadian medical students were set by using sequential rounds of revision by three consecutive groups of stakeholders and evaluation experts. Structured communication was guided by a modified Delphi technique. The feasibility/consequence models of these EPAs were then assessed by tracking their completion by the graduating class of 2021. RESULTS: The threshold-setting process resulted in set EPA achievement levels ranging from 1 to 8 across the 12 AFMC EPAs. Estimates were stable after the first round for 9 of 12 EPAs. 96.27% of EPAs were successfully completed by clerkship students despite the shortened clinical period. Feasibility was predicted by the slowing rate of EPA accumulation overtime during the clerkship. CONCLUSION: The process described led to consensus on EPA achievement thresholds. Successful completion of the assigned thresholds was feasible within the shortened clerkship.[Box: see text].

The Validity of MCAT Scores in Predicting Students' Performance and Progress in Medical School: Results From a Multisite Study.

PURPOSE: This is the first multisite investigation of the validity of scores from the current version of the Medical College Admission Test (MCAT) in clerkship and licensure contexts. It examined the predictive validity of MCAT scores and undergraduate grade point averages (UGPAs) for performance in preclerkship and clerkship courses and on the United States Medical Licensing Examination Step 1 and Step 2 Clinical Knowledge examinations. It also studied students' progress in medical school. METHOD: Researchers examined data from 17 U.S. and Canadian MD-granting medical schools for 2016 and 2017 entrants who volunteered for the research and applied with scores from the current MCAT exam. They also examined data for all U.S. medical schools for 2016 and 2017 entrants to regular-MD programs who applied with scores from the current exam. Researchers conducted linear and logistic regression analyses to determine whether MCAT total scores added value beyond UGPAs in predicting medical students' performance and progress. Importantly, they examined the comparability of prediction by sex, race and ethnicity, and socioeconomic status. RESULTS: Researchers reported medium to large correlations between MCAT total scores and medical student outcomes. Correlations between total UGPAs and medical student outcomes were similar but slightly lower. When MCAT scores and UGPAs were used together, they predicted student performance and progress better than either alone. Despite differences in average MCAT scores and UGPAs between students who self-identified as White or Asian and those from underrepresented racial and ethnic groups, predictive validity results were comparable. The same was true for students from different socioeconomic backgrounds, and for males and females. CONCLUSIONS: These data demonstrate that MCAT scores add value to the prediction of medical student performance and progress and that applicants from different backgrounds who enter medical school with similar ranges of MCAT scores and UGPAs perform similarly in the curriculum.

Can we predict failure in licensure exams from medical students' undergraduate academic performance?

BACKGROUND: In 2015, the Medical Council of Canada increased the minimum pass level for the Medical Council of Canada Qualifying Examination Part I, and students had a higher rate of failure than in previous years. The purpose of this study was to predict students at an increased odds of examination failure to allow for early, targeted interventions. METHODS: We divided our dataset into a derivation cohort and two validation cohorts and used multiple logistic regression to predict licensing examination failure. We then performed receiver operating characteristics and a sensitivity analysis using different cutoffs for explanatory variables to identify the cutoff threshold with the best predictive value at identifying students at increased odds of failure. RESULTS: After multivariate analysis, only pre-clerkship GPA was a significant independent predictor of failure (OR 0.76, 95% CI [0.66, 0.88], p < 0.001). The probability of failure increased steeply when the pre-clerkship GPA fell below 80% and 76% was found to be the most efficient cutoff for predicting failure (OR 9.37, 95% CI [3.08, 38.41]). CONCLUSIONS: Pre-clerkship performance can predict students at increased odds of licensing examination failure. Further studies are needed to explore whether early interventions for at-risk students alter their examination performance.

CONTEXTE: En 2015, le Conseil médical du Canada a resserré les exigences de réussite à l'examen d'aptitude du Conseil médical du Canada, partie I, entraînant un taux d'échec plus élevé que les années précédentes. L'objectif de cette étude était de détecter les étudiants ayant de plus grande probabilité d'échec à l'examen afin de permettre des interventions ciblées en temps utile. MÉTHODES: Nous avons comparé les données d'une cohorte de dérivation et deux cohortes de validation et nous avons utilisé la régression logistique multiple pour prédire l'échec à l'examen d'aptitude. Nous avons ensuite effectué une analyse de la fonction d'efficacité du récepteur et une analyse de sensibilité en utilisant différents seuils pour les variables explicatives afin de déterminer la meilleure valeur prédictive seuil pour cibler une forte possibilité d'échec chez les étudiants. RÉSULTATS: L'analyse multivariée a révélé que seule la moyenne générale des étudiants était un prédicteur indépendant significatif de l'échec (OR 0.76, 95 % CI [0.66, 0.88], p < 0.001). La probabilité d'échec augmentait fortement lorsque l'indice de moyenne générale tombait en dessous de 80 %. Le seuil le plus efficace pour prédire l'échec s'est avéré être 76 % (OR 9,37, 95 % CI [3,08, 38,41]). CONCLUSIONS: Les résultats scolaires des étudiants en médecine constituent un indicateur de risque d'échec à l'examen d'aptitude. Des études supplémentaires sont nécessaires pour vérifier si une intervention précoce auprès des étudiants à risque peut améliorer leurs résultats à l'examen.

The Validity of Scores From the New MCAT Exam in Predicting Student Performance: Results From a Multisite Study.

PURPOSE: The new Medical College Admission Test (MCAT) was introduced in April 2015. This report presents findings from the first study of the validity of scores from the new MCAT exam in predicting student performance in the first year of medical school (M1). METHOD: The authors analyzed data from the national population of 2016 matriculants with scores from the new MCAT exam (N = 7,970) and the sample of 2016 matriculants (N = 955) from 16 medical schools who volunteered to participate in the validity research. They examined correlations of students' MCAT total scores and total undergraduate grade point averages (UGPAs), alone and together, with their summative performance in M1, and the success rate of students with different MCAT scores in their on-time progression to the second year of medical school (M2). They assessed whether MCAT scores provided comparable prediction of performance in M1 by students' race/ethnicity, socioeconomic background, and gender. RESULTS: Correlations of MCAT scores with summative performance in M1 ranged from medium to large. Although MCAT scores and UGPAs provided similar prediction of performance in M1, using both metrics provided better prediction than either alone. Additionally, students with a wide range of MCAT scores progressed to M2 on time. Finally, MCAT scores provided comparable prediction of performance in M1 for students from different sociodemographic backgrounds. CONCLUSIONS: This study provides early evidence that scores from the new MCAT exam predict student performance in M1. Future research will examine the validity of MCAT scores in predicting performance in later years.

Choosing Our Own Pathway to Competency-Based Undergraduate Medical Education.

After many years in the making, an increasing number of postgraduate medical education (PGME) training programs in North America are now adopting a competency-based medical education (CBME) framework based on entrustable professional activities (EPAs) that, in turn, encompass a larger number of competencies and training milestones. Following the lead of PGME, CBME is now being incorporated into undergraduate medical education (UME) in an attempt to improve integration across the medical education continuum and to facilitate a smooth transition from clerkship to residency by ensuring that all graduates are ready for indirect supervision of required EPAs on day one of residency training. The Association of Faculties of Medicine of Canada recently finalized its list of 12 EPAs, which closely parallels the list of 13 EPAs published earlier by the Association of American Medical Colleges, and defines the "core" EPAs that are an expectation of all medical school graduates.In this article, the authors focus on important, practical considerations for the transition to CBME that they feel have not been adequately addressed in the existing literature. They suggest that the transition to CBME should not threaten diversity in UME or require a major curricular upheaval. However, each UME program must make important decisions that will define its version of CBME, including which terminology to use when describing the construct being evaluated, which rating tools and raters to include in the assessment program, and how to make promotion decisions based on all of the available data on EPAs.

The grades that clinical teachers give students modifies the grades they receive.

Prior studies have shown a correlation between the grades students receive and how they rate their teacher in the classroom. In this study, the authors probe this association on clinical rotations and explore potential mechanisms. All In-Training Evaluation Reports (ITERs) for students on mandatory clerkship rotations from April 1, 2013 to January 31, 2015 were matched with the corresponding student's rating of their teacher (SRT). The date and time that ITERs and SRTs were submitted was used to divide SRTs into those submitted before versus after the corresponding ITER was submitted. Multilevel, mixed effects linear regression was used to examine the association between SRT, ITER rating, and whether the ITER was submitted before or after SRT. Of 2373 paired evaluations, 1098 (46.3%) SRT were submitted before the teacher had submitted the ITER. There was a significant interaction between explanatory variables: when ITER ratings had not yet been submitted, the regression coefficient for this association was 0.25 (95% confidence interval [0.17, 0.33], p < 0.001), whereas the regression coefficient was significantly higher when ITER ratings were submitted prior to SRT (0.40 [0.31, 0.49], p < 0.001). Finding an association between SRT and ITER when students do not know their ITER ratings suggests that SRTs can capture attributes of effective teaching, but the effect modification when students have access to their ITER rating supports grade satisfaction bias. Further studies are needed to explain the mechanism of grade satisfaction and to identify other biases that may impact the validity of SRT.

How teachers can help learners build storage and retrieval strength.

AIM: To be an effective teacher, content expertise is necessary but alone does not guarantee optimal learning outcomes for students. In this article, the authors discuss ways in which medical teachers can shape the learning of their students and enable them to become more efficient and effective learners. METHODS: Using Bjork and Bjork's new theory of disuse as their framework, the authors discuss strategies to improve storage strength of to-be-learned information and strategies to improve retrieval strength of learned information. RESULTS: Strategies to improve storage strength include optimizing cognitive load, providing causal explanations, and giving effective feedback. Strategies to improve retrieval strength include situated cognition and various types of retrieval practice. CONCLUSIONS: Adopting these teaching strategies should hopefully help teachers improve the learning outcomes of their students, but there is still a need for further research into the science of learning and the science of instruction, including comparative effectiveness of different teaching strategies and how best to translate findings from the psychology literature into medical education.

Weighing the cost of educational inflation in undergraduate medical education.

Despite the fact that the length of medical school training has remained stable for many years, the expectations of graduating medical students (and the schools that train them) continue to increase. In this Reflection, the authors discuss motives for educational inflation and suggest that these are likely innocent, well-intentioned, and subconscious-and include both a propensity to increase expectations of ourselves and others over time, and a reluctance to reduce training content and expectations. They then discuss potential risks of educational inflation, including reduced emphasis on core knowledge and clinical skills, and adverse effects on the emotional, psychological, and financial wellbeing of students. While acknowledging the need to change curricula to improve learning and clinical outcomes, the authors proffer that it is naïve to assume that we can inflate educational expectations at no additional cost. They suggest that before implementing and/or mandating change, we should consider of all the costs that medical schools and students might incur, including opportunity costs and the impact on the emotional and financial wellbeing of students. They propose a cost-effectiveness framework for medical education and advocate prioritization of interventions that improve learning outcomes with no additional costs or are cost-saving without adversely impacting learning outcomes. When there is an additional cost for improved learning outcomes or a decline in learning outcomes as a result of cost saving interventions, they suggest careful consideration and justification of this trade-off. And when there are neither improved learning outcomes nor cost savings they recommend resisting the urge to change.

Making progress in the ethical treatment of medical trainees.

There is an inherent conflict within clinician educators as we balance the roles of healthcare provider to patients in need of care with that of educator of learners in need of teaching. In this essay we use Beauchamp and Childress' principles of biomedical ethics as a framework to compare the relationship that clinician educators have with their patients and their learners, and suggest that while we typically apply ethical principles when addressing the needs of our patients, these principles are frequently lacking in our interactions with learners. This dichotomy reflects a person-by-situation interaction that may be partly explained by the expectations of the regulatory bodies that define how clinicians should interact with patients and how educators should interact with learners. The result is that we may fall short in applying respect for autonomy, beneficence/nonmaleficence, and justice when addressing the needs of our learners. Fortunately there are ways in which we can incorporate these ethical principles into our interactions with learners while still adhering to accreditation standards and institutional policy. These include flipped classrooms and simulated learning experiences, incorporating aspects of instructional design that have been shown to improve learning outcomes, providing additional resources to learners with greater needs, and organizing training curricula around entrustable professional activities. Although the consistent application of ethical principles with all learners during all learning experiences is likely unachievable, we can, and should, move towards more ethical treatment of our learners.

Long-term persistence with injectable therapy in relapsing-remitting multiple sclerosis: an 18-year observational cohort study.

Disease modifying therapies (DMTs) reduce the frequency of relapses and accumulation of disability in multiple sclerosis (MS). Long-term persistence with treatment is important to optimize treatment benefit. This long-term, cohort study was conducted at the Calgary MS Clinic. All consenting adults with relapsing-remitting MS who started either glatiramer acetate (GA) or interferon-ß 1a/1b (IFN-ß) between January 1st, 1996 and July 1st, 2011 were included. Follow-up continued to February 1st, 2014. Time-to-discontinuation of the initial and subsequently-prescribed DMTs (switches) was analysed using Kaplan-Meier survival analyses. Group differences were compared using log-rank tests and multivariable Cox regression models. Analysis included 1471 participants; 906 were initially prescribed GA and 565 were initially prescribed IFN-ß. Follow-up information was available for 87%; 29 (2%) were lost to follow-up and 160 (11%) moved from Southern Alberta while still using DMT. Median time-to-discontinuation of all injectable DMTs was 11.1 years. Participants with greater disability at treatment initiation, those who started treatment before age 30, and those who started between 2006 and 2011 were more likely to discontinue use of all injectable DMTs. Median time-to-discontinuation of the initial DMT was 8.6 years. Those initially prescribed GA remained on treatment longer. Of 610 participants who discontinued injectable DMT, 331 (54%) started an oral DMT, or a second-line DMT, or resumed injectable DMT after 90 days. Persistence with injectable DMTs was high in this long-term population-based study. Most participants who discontinued injectable DMT did not remain untreated. Further research is required to understand treatment outcomes and outcomes after stopping DMT.

Neurologic disorders associated with weight lifting and bodybuilding.

Weight lifting and other forms of strength training are becoming more common because of an increased awareness of the need to maintain individual physical fitness. Emergency room data indicate that injuries caused by weight training have become more universal over time, likely because of increased participation rates. Neurologic injuries can result from weight lifting and related practices. Although predominantly peripheral nervous system injuries have been described, central nervous system disease may also occur. This article illustrates the types of neurologic disorders associated with weight lifting.

Sphenoid sinus mucocele--rupture causing brainstem inflammation and stroke.

BACKGROUND: Posterior rupture of a sphenoid sinus mucocele is a rare cause of brainstem injury. METHODS: Case report. RESULTS: A healthy young woman with a history of prior surgical excision of nasal polyps presented with a headache and evolving neurologic symptoms. The clinical presentation and imaging studies were consistent with a posterior rupture of a large sphenoid mucocele with subsequent inflammatory changes in the brainstem and evidence of brainstem stroke. Relatively rapid recovery coincided temporally with the resolution of local inflammation. Longer-term recovery continued over months, as expected following stroke. CONCLUSIONS: Posterior rupture of a sphenoid sinus mucocele may cause brainstem injury by multiple mechanisms.

Neurologic disorders associated with weight lifting and bodybuilding.

Weight lifting and other forms of strength training are becoming more common because of an increased awareness of the need to maintain individual physical fitness. Emergency room data indicate that injuries caused by weight training have become more universal over time, likely because of increased participation rates. Neurologic injuries can result from weight lifting and related practices. Although predominantly peripheral nervous system injuries have been described, central nervous system disease may also occur. This article illustrates the types of neurologic disorders associated with weight lifting.

Acute necrotizing encephalopathy in caucasian children: two cases and review of the literature.

Acute necrotizing encephalopathy is a fulminant neurologic disease seen predominantly in Japan and Taiwan. We present two cases diagnosed at a Canadian center within the same year in Caucasian children. Both were previously well, developed an acute viral illness with fever and vomiting, and progressed to brain death within 2 to 4 days. Neuroimaging and postmortem examination demonstrated the unique features of bilateral and severe necrosis of deep gray- and subcortical white-matter structures. The first case was associated with extensive, but transient, hepatic involvement, recent varicella and rotavirus infections, and detailed metabolic studies, including mitochondrial functional analysis, were normal. The second case tested positive for influenza A infection, whereas evidence of liver damage was lacking. Both children demonstrated early lymphopenia and myocardial necrosis, two features not previously associated with acute necrotizing encephalopathy. These cases are unique in their occurrence in non-Japanese children and are among the first published reports in Canada.