Can we predict failure in licensure exams from medical students' undergraduate academic performance?

BACKGROUND: In 2015, the Medical Council of Canada increased the minimum pass level for the Medical Council of Canada Qualifying Examination Part I, and students had a higher rate of failure than in previous years. The purpose of this study was to predict students at an increased odds of examination failure to allow for early, targeted interventions. METHODS: We divided our dataset into a derivation cohort and two validation cohorts and used multiple logistic regression to predict licensing examination failure. We then performed receiver operating characteristics and a sensitivity analysis using different cutoffs for explanatory variables to identify the cutoff threshold with the best predictive value at identifying students at increased odds of failure. RESULTS: After multivariate analysis, only pre-clerkship GPA was a significant independent predictor of failure (OR 0.76, 95% CI [0.66, 0.88], p < 0.001). The probability of failure increased steeply when the pre-clerkship GPA fell below 80% and 76% was found to be the most efficient cutoff for predicting failure (OR 9.37, 95% CI [3.08, 38.41]). CONCLUSIONS: Pre-clerkship performance can predict students at increased odds of licensing examination failure. Further studies are needed to explore whether early interventions for at-risk students alter their examination performance.

CONTEXTE: En 2015, le Conseil médical du Canada a resserré les exigences de réussite à l'examen d'aptitude du Conseil médical du Canada, partie I, entraînant un taux d'échec plus élevé que les années précédentes. L'objectif de cette étude était de détecter les étudiants ayant de plus grande probabilité d'échec à l'examen afin de permettre des interventions ciblées en temps utile. MÉTHODES: Nous avons comparé les données d'une cohorte de dérivation et deux cohortes de validation et nous avons utilisé la régression logistique multiple pour prédire l'échec à l'examen d'aptitude. Nous avons ensuite effectué une analyse de la fonction d'efficacité du récepteur et une analyse de sensibilité en utilisant différents seuils pour les variables explicatives afin de déterminer la meilleure valeur prédictive seuil pour cibler une forte possibilité d'échec chez les étudiants. RÉSULTATS: L'analyse multivariée a révélé que seule la moyenne générale des étudiants était un prédicteur indépendant significatif de l'échec (OR 0.76, 95 % CI [0.66, 0.88], p < 0.001). La probabilité d'échec augmentait fortement lorsque l'indice de moyenne générale tombait en dessous de 80 %. Le seuil le plus efficace pour prédire l'échec s'est avéré être 76 % (OR 9,37, 95 % CI [3,08, 38,41]). CONCLUSIONS: Les résultats scolaires des étudiants en médecine constituent un indicateur de risque d'échec à l'examen d'aptitude. Des études supplémentaires sont nécessaires pour vérifier si une intervention précoce auprès des étudiants à risque peut améliorer leurs résultats à l'examen.

Summative Assessment of Interprofessional "Collaborative Practice" Skills in Graduating Medical Students: A Validity Argument.

PURPOSE: To describe how the authors developed an objective structured clinical examination (OSCE) station to assess aspects of collaborative practice competency and how they then assessed validity using Kane's framework. METHOD: After piloting the collaborative practice OSCE station in 2015 and 2016, this was introduced at the Cumming School of Medicine in 2017. One hundred fifty-five students from the class of 2017 and 22 students from the class of 2018 participated. To create a validity argument, the authors used Kane's framework that views the argument for validity as 4 sequential inferences on the validity of scoring, generalization, extrapolation, and implications, RESULTS: Scoring validity is supported by psychometric analysis of checklist items and the fact that the contribution of rater specificity to students' ratings was similar to OSCE stations assessing clinical skills alone. The claim of validity of generalization is backed by structural equation modeling and confirmatory factor analysis that identified 5 latent variables, including 3 related to collaborative practice ("provides an effective handover," "provides mutual support," and "shares their mental model"). Validity of extrapolation is argued based upon the correlation between the rating for "shares their mental model" and the rating on in-training evaluations for "relationship with other members of the health care team," in addition to the association between performance on the collaborative practice OSCE station and the subsequent rating of performance during residency. Finally, validity of implications is supported by the fact that pass/fail decisions on the collaborative practice station were similar to other stations and by the observation that ratings on different aspects of collaborative practice associate with pass/fail decisions. CONCLUSIONS: Based upon the validity argument presented, the authors posit that this tool can be used to assess the collaborative practice competence of graduating medical students and the adequacy of training in collaborative practice.

Quantifying two-dimensional and three-dimensional stereoscopic learning in anatomy using electroencephalography.

Advances in computer visualization enabling both 2D and 3D representation have generated tools to aid perception of spatial relationships and provide a new forum for instructional design. A key knowledge gap is the lack of understanding of how the brain neurobiologically processes and learns from spatially presented content, and new quantitative variables are required to address this gap. The objective of this study was to apply quantitative neural measures derived from electroencephalography (EEG) to examine stereopsis in anatomy learning by comparing mean amplitude changes in N250 (related to object recognition) and reward positivity (related to responding to feedback) event related to potential components using a reinforcement-based learning paradigm. Health sciences students (n = 61) learned to identify and localize neuroanatomical structures using 2D, 3D, or a combination of models while EEG and behavioral (accuracy) data were recorded. Participants learning using 3D models had a greater object recognition (N250 amplitude) compared to those who learned from 2D models. Based on neurological results, interleaved learning incorporating both 2D and 3D models provided an advantage in learning, retention, and transfer activities represented by decreased reward positivity amplitude. Behavioral data did not have the same sensitivity as neural data for distinguishing differences in learning with and without stereopsis in these learning activities. Measuring neural activity reveals new insights in applied settings for educators to consider when incorporating stereoscopic models in the design of learning interventions.

How can we reduce bias during an academic assessment reappraisal?

Aims: To describe potential sources of bias during an academic assessment reappraisal and ways to mitigate these. Methods: We describe why the typical scenario of an academic assessment reappraisal - where committee members are asked to weigh contrasting accounts of past events that they did not witness, and to rate elusive constructs, such as "fairness" - is prone to multiple types of bias, including attribute substitution, default bias, confirmation bias, and impact bias. We also discuss how increased awareness of sources of bias and of debiasing strategies can improve the validity of decision making. Results: Strategies that can reduce bias in reappraisal include clearly articulating and focusing on the reappraisal question (did bias cause a wrong decision to be made?), educating those involved in the reappraisal of the types of bias that frequently occur in teaching and assessment (including biases that they themselves may introduce to the reappraisal), and ensuring that those involved in the reappraisal contribute equally to making decisions and recommendation. Conclusions: All academic assessments of students, particularly those that involve subjective ratings of performance, are prone to bias, which threatens the integrity of the assessment process. Given the high stakes of academic assessments, we feel that each medical school should have a process for assessment reappraisal that reduces, rather than compounds, the likelihood of wrong assessment decisions.

Does the Medical College Admission Test (MCAT) predict licensing examination performance in the Canadian context?

BACKGROUND: Research on the predictive validity of the Medical College Admissions Test (MCAT) on licensing examination performance is varied in its conclusions, with only a few studies examining this relationship in a Canadian context. We assessed the predictive validity of the MCAT on successful performance on the Medical Council of Canada Qualifying Examination (MCCQE) Part 1 by students attending the Cumming School of Medicine. METHODS: Prospective data were collected on MCAT score and sub-section scores, MCCQE decision, multiple mini interview (MMI) performance, gender, and age. The cohort was divided into a derivation cohort (2013 and 2014) and validation cohort (2015 and 2016). Students were dichotomized into pass or fail on MCCQE. Multiple logistic regression in which our dependent variable was MCCQE Part I examination success at the first attempt was used, and potential explanatory variables were age, gender, MCAT total score, and sub-scores for the biological sciences (MCAT-BS), physical sciences, and verbal reasoning, GPA, and MMI ratings. RESULTS: For the derivation cohort MCAT-BS was associated with success on the MCCQE Part I. The odds ratio for this association of 1.37 (95% confidence interval [1.01, 1.85], p = 0.04). When we applied the MCAT-BS to our validation cohort the odds ratio of MCCQE Part I examination success was 1.42 [1.10, 1.83], p = 0.007) and the area under the ROC curve was 0.66 [0.54, 0.79]). CONCLUSION: The MCAT-BS predicted successful performance on the MCCQE Part 1 Examination in the Canadian setting.

Choosing Our Own Pathway to Competency-Based Undergraduate Medical Education.

After many years in the making, an increasing number of postgraduate medical education (PGME) training programs in North America are now adopting a competency-based medical education (CBME) framework based on entrustable professional activities (EPAs) that, in turn, encompass a larger number of competencies and training milestones. Following the lead of PGME, CBME is now being incorporated into undergraduate medical education (UME) in an attempt to improve integration across the medical education continuum and to facilitate a smooth transition from clerkship to residency by ensuring that all graduates are ready for indirect supervision of required EPAs on day one of residency training. The Association of Faculties of Medicine of Canada recently finalized its list of 12 EPAs, which closely parallels the list of 13 EPAs published earlier by the Association of American Medical Colleges, and defines the "core" EPAs that are an expectation of all medical school graduates.In this article, the authors focus on important, practical considerations for the transition to CBME that they feel have not been adequately addressed in the existing literature. They suggest that the transition to CBME should not threaten diversity in UME or require a major curricular upheaval. However, each UME program must make important decisions that will define its version of CBME, including which terminology to use when describing the construct being evaluated, which rating tools and raters to include in the assessment program, and how to make promotion decisions based on all of the available data on EPAs.

Are we failing to "build on the scientific basis of medicine?"

In this article, we question if and why the Canadian national medical education organizations have failed to introduce or promote changes that compel or encourage Canadian medical schools to heed the recommendation from the Future of Medical Education in Canada report to "build on the scientific basis of medicine." We end by offering suggestions on how these organizations could help Canadian medical schools build in the scientific basis of medicine.

The grades that clinical teachers give students modifies the grades they receive.

Prior studies have shown a correlation between the grades students receive and how they rate their teacher in the classroom. In this study, the authors probe this association on clinical rotations and explore potential mechanisms. All In-Training Evaluation Reports (ITERs) for students on mandatory clerkship rotations from April 1, 2013 to January 31, 2015 were matched with the corresponding student's rating of their teacher (SRT). The date and time that ITERs and SRTs were submitted was used to divide SRTs into those submitted before versus after the corresponding ITER was submitted. Multilevel, mixed effects linear regression was used to examine the association between SRT, ITER rating, and whether the ITER was submitted before or after SRT. Of 2373 paired evaluations, 1098 (46.3%) SRT were submitted before the teacher had submitted the ITER. There was a significant interaction between explanatory variables: when ITER ratings had not yet been submitted, the regression coefficient for this association was 0.25 (95% confidence interval [0.17, 0.33], p < 0.001), whereas the regression coefficient was significantly higher when ITER ratings were submitted prior to SRT (0.40 [0.31, 0.49], p < 0.001). Finding an association between SRT and ITER when students do not know their ITER ratings suggests that SRTs can capture attributes of effective teaching, but the effect modification when students have access to their ITER rating supports grade satisfaction bias. Further studies are needed to explain the mechanism of grade satisfaction and to identify other biases that may impact the validity of SRT.

How teachers can help learners build storage and retrieval strength.

AIM: To be an effective teacher, content expertise is necessary but alone does not guarantee optimal learning outcomes for students. In this article, the authors discuss ways in which medical teachers can shape the learning of their students and enable them to become more efficient and effective learners. METHODS: Using Bjork and Bjork's new theory of disuse as their framework, the authors discuss strategies to improve storage strength of to-be-learned information and strategies to improve retrieval strength of learned information. RESULTS: Strategies to improve storage strength include optimizing cognitive load, providing causal explanations, and giving effective feedback. Strategies to improve retrieval strength include situated cognition and various types of retrieval practice. CONCLUSIONS: Adopting these teaching strategies should hopefully help teachers improve the learning outcomes of their students, but there is still a need for further research into the science of learning and the science of instruction, including comparative effectiveness of different teaching strategies and how best to translate findings from the psychology literature into medical education.

The application of reward learning in the real world: Changes in the reward positivity amplitude reflect learning in a medical education context.

Evidence ranging from behavioural adaptations to neurocognitive theories has made significant advances into our understanding of feedback-based learning. For instance, over the past twenty years research using electroencephalography has demonstrated that the amplitude of a component of the human event-related brain potential - the reward positivity - appears to change with learning in a manner predicted by reinforcement learning theory (Holroyd and Coles, 2002; Sutton and Barto, 1998). However, while the reward positivity (also known as the feedback related negativity) is well studied, whether the component reflects an underlying learning process or whether it is simply sensitive to feedback evaluation is still unclear. Here, we sought to provide support that the reward positivity is reflective of an underlying learning process and further we hoped to demonstrate this in a real-world medical education context. In the present study, students with no medical training viewed a series of patient cards that contained ten physiological readings relevant for diagnosing liver and biliary disease types, selected the most appropriate diagnostic classification, and received feedback as to whether their decisions were correct or incorrect. Our behavioural results revealed that our participants were able to learn to diagnose liver and biliary disease types. Importantly, we found that the amplitude of the reward positivity diminished in a concomitant manner with the aforementioned behavioural improvements. In sum, our data support theoretical predictions (e.g., Holroyd and Coles, 2002), suggest that the reward positivity is an index of a neural learning system, and further validate that this same system is involved in learning across a wide range of contexts.

Are Female Applicants Rated Higher Than Males on the Multiple Mini-Interview? Findings From the University of Calgary.

PURPOSE: The multiple mini-interview (MMI) improves reliability and validity of medical school interviews, and many schools have introduced this in an attempt to select individuals more skilled in communication, critical thinking, and ethical decision making. But every change in the admissions process may produce unintended consequences, such as changing intake demographics. In this article, two studies exploring gender differences in MMI ratings are reported. METHOD: Cumulative meta-analysis was used to compare MMI ratings for female and male applicants to the University of Calgary Cumming School of Medicine between 2010 and 2014. Multiple linear regression was then performed to explore gender differences in MMI ratings after adjusting for other variables, followed by a sensitivity analysis of the impact of varying the weight given to MMI ratings on the odds of females being ranked in the top 150 applicants for 2014. RESULTS: Females were rated higher than male applicants (standardized mean difference 0.21, 95% CI [0.11, 0.30], P < .001). After adjusting for other explanatory variables, there was a positive association between female applicant and MMI rating (regression coefficient 0.23 [0.14, 0.33], P < .001). Increasing weight assigned to MMI ratings was associated with increased odds of females being ranked in the top 150 applicants. CONCLUSIONS: In this single-center study, females were rated higher than males on the MMI, and the odds of a female applicant being offered a position increased as more weight was given to MMI ratings. Further studies are needed to confirm and explain gender differences in MMI ratings.

Cognitive load imposed by ultrasound-facilitated teaching does not adversely affect gross anatomy learning outcomes.

Ultrasonography is increasingly used in medical education, but its impact on learning outcomes is unclear. Adding ultrasound may facilitate learning, but may also potentially overwhelm novice learners. Based upon the framework of cognitive load theory, this study seeks to evaluate the relationship between cognitive load associated with using ultrasound and learning outcomes. The use of ultrasound was hypothesized to facilitate learning in anatomy for 161 novice first-year medical students. Using linear regression analyses, the relationship between reported cognitive load on using ultrasound and learning outcomes as measured by anatomy laboratory examination scores four weeks after ultrasound-guided anatomy training was evaluated in consenting students. Second anatomy examination scores of students who were taught anatomy with ultrasound were compared with historical controls (those not taught with ultrasound). Ultrasound's perceived utility for learning was measured on a five-point scale. Cognitive load on using ultrasound was measured on a nine-point scale. Primary outcome was the laboratory examination score (60 questions). Learners found ultrasound useful for learning. Weighted factor score on "image interpretation" was negatively, but insignificantly, associated with examination scores [F (1,135) = 0.28, beta = -0.22; P = 0.61]. Weighted factor score on "basic knobology" was positively and insignificantly associated with scores; [F (1,138) = 0.27, beta = 0.42; P = 0.60]. Cohorts exposed to ultrasound had significantly higher scores than historical controls (82.4% ± SD 8.6% vs. 78.8% ± 8.5%, Cohen's d = 0.41, P < 0.001). Using ultrasound to teach anatomy does not negatively impact learning and may improve learning outcomes. Anat Sci Educ 10: 144-151. © 2016 American Association of Anatomists.

Weighing the cost of educational inflation in undergraduate medical education.

Despite the fact that the length of medical school training has remained stable for many years, the expectations of graduating medical students (and the schools that train them) continue to increase. In this Reflection, the authors discuss motives for educational inflation and suggest that these are likely innocent, well-intentioned, and subconscious-and include both a propensity to increase expectations of ourselves and others over time, and a reluctance to reduce training content and expectations. They then discuss potential risks of educational inflation, including reduced emphasis on core knowledge and clinical skills, and adverse effects on the emotional, psychological, and financial wellbeing of students. While acknowledging the need to change curricula to improve learning and clinical outcomes, the authors proffer that it is naïve to assume that we can inflate educational expectations at no additional cost. They suggest that before implementing and/or mandating change, we should consider of all the costs that medical schools and students might incur, including opportunity costs and the impact on the emotional and financial wellbeing of students. They propose a cost-effectiveness framework for medical education and advocate prioritization of interventions that improve learning outcomes with no additional costs or are cost-saving without adversely impacting learning outcomes. When there is an additional cost for improved learning outcomes or a decline in learning outcomes as a result of cost saving interventions, they suggest careful consideration and justification of this trade-off. And when there are neither improved learning outcomes nor cost savings they recommend resisting the urge to change.

Are we at risk of groupthink in our approach to teamwork interventions in health care?

CONTEXT: The incidence of medical error, adverse clinical events and poor quality health care is unacceptably high and there are data to suggest that poor coordination of care, or teamwork, contributes to adverse outcomes. So, can we assume that increased collaboration in multidisciplinary teams improves performance and health care outcomes for patients? METHODS: In this essay, the authors discuss some reasons why we should not presume that collective decision making leads to better decisions and collaborative care results in better health care outcomes. RESULTS: Despite an exponential increase in interventions designed to improve teamwork and interprofessional education (IPE), we are still lacking good quality data on whether these interventions improve important outcomes. There are reasons why some of the components of 'effective teamwork', such as shared mental models, team orientation and mutual trust, could impair delivery of health care. For example, prior studies have found that brainstorming results in fewer ideas rather than more, and hinders rather than helps productivity. There are several possible explanations for this effect, including 'social loafing' and cognitive overload. Similarly, attributes that improve cohesion within groups, such as team orientation and mutual trust, may increase the risk of 'groupthink' and group conformity bias, which may lead to poorer decisions. CONCLUSIONS: In reality, teamwork and IPE are not inherently good, bad or neutral; instead, as with any intervention, their effect is modified by the persons involved, the situation and the interaction between persons and situation. Thus, rather than assume better outcomes with teamwork and IPE interventions, as clinicians and educators we must demonstrate that our interventions improve the delivery of health care.

The attributes of an effective teacher differ between the classroom and the clinical setting.

Most training programs use learners' subjective ratings of their teachers as the primary measure of teaching effectiveness. In a recent study we found that preclinical medical students' ratings of classroom teachers were associated with perceived charisma and physical attractiveness of the teacher, but not intellect. Here we explored whether the relationship between these variables and teaching effectiveness ratings holds in the clinical setting. We asked 27 Internal Medicine residents to rate teaching effectiveness of ten teachers with whom they had worked on a clinical rotation, in addition to rating each teacher's clinical skills, physical attractiveness, and charisma. We used linear regression to study the association between these explanatory variables and teaching effectiveness ratings. We found no association between rating of physical attractiveness and teaching effectiveness. Clinical skill and charisma were independently associated with rating of teaching effectiveness (regression coefficients [95 % confidence interval] 0.73 [0.60, 0.85], p < 0.001 and 0.12 [0.01, 0.23], p = 0.03, respectively). The variables associated with effectiveness of classroom and clinical teachers differ, suggesting context specificity in teaching effectiveness ratings. Context specificity may be explained by differences in the exposure that learners have to teachers in the classroom versus clinical setting-so that raters in the clinical setting may base ratings upon observed behaviours rather than stereotype data. Alternatively, since subjective ratings of teaching effectiveness inevitably incorporate learners' context-specific needs, the attributes that make a teacher effective in one context may not meet the needs of learners in a different context.

Making progress in the ethical treatment of medical trainees.

There is an inherent conflict within clinician educators as we balance the roles of healthcare provider to patients in need of care with that of educator of learners in need of teaching. In this essay we use Beauchamp and Childress' principles of biomedical ethics as a framework to compare the relationship that clinician educators have with their patients and their learners, and suggest that while we typically apply ethical principles when addressing the needs of our patients, these principles are frequently lacking in our interactions with learners. This dichotomy reflects a person-by-situation interaction that may be partly explained by the expectations of the regulatory bodies that define how clinicians should interact with patients and how educators should interact with learners. The result is that we may fall short in applying respect for autonomy, beneficence/nonmaleficence, and justice when addressing the needs of our learners. Fortunately there are ways in which we can incorporate these ethical principles into our interactions with learners while still adhering to accreditation standards and institutional policy. These include flipped classrooms and simulated learning experiences, incorporating aspects of instructional design that have been shown to improve learning outcomes, providing additional resources to learners with greater needs, and organizing training curricula around entrustable professional activities. Although the consistent application of ethical principles with all learners during all learning experiences is likely unachievable, we can, and should, move towards more ethical treatment of our learners.

Hemispheric activation differences in novice and expert clinicians during clinical decision making.

Clinical decision making requires knowledge, experience and analytical/non-analytical types of decision processes. As clinicians progress from novice to expert, research indicates decision-making becomes less reliant on foundational biomedical knowledge and more on previous experience. In this study, we investigated how knowledge and experience were reflected in terms of differences in neural areas of activation. Novice and expert clinicians diagnosed simple or complex (easy, hard) cases while functional magnetic resonance imaging (fMRI) data were collected. Our results highlight key differences in the neural areas activated in novices and experts during the clinical decision-making process. fMRI data were collected from ten second year medical students (novices) and ten practicing gastroenterologists (experts) while they diagnosed sixteen (eight easy and eight hard) clinical cases via multiple-choice questions. Behavioral data were collected for diagnostic accuracy (correct/incorrect diagnosis) and time taken to assign a clinical diagnosis. Two analyses were performed with the fMRI data. First, data from easy and hard cases were compared within respective groups (easy > hard, hard > easy). Second, neural differences between novices and experts (novice > expert, expert > novice) were assessed. Experts correctly diagnosed more cases than novices and made their diagnoses faster than novices on both easy and hard cases (all p's < 0.05). Time taken to diagnose hard cases took significantly longer for both novices and experts. While similar neural areas were activated in both novices and experts during the decision making process, we identified significant hemispheric activation differences between novice and expert clinicians when diagnosing hard clinical cases. Specifically, novice clinicians had greater activations in the left anterior temporal cortex and left ventral lateral prefrontal cortex whereas expert clinicians had greater activations in the right dorsal lateral, right ventral lateral, and right parietal cortex. Hemispheric differences in activation were not observed between novices and experts while diagnosing easy clinical cases. While clinical decision-making engaged the prefrontal cortex (PFC) in both novices and experts, interestingly we observed expertise related differences in the regions and hemispheres of PFC activation between these groups for hard clinical cases. Specifically, in novices we observed activations in left hemisphere neural regions associated with factual rule-based knowledge, whereas in experts we observed right hemisphere activation in neural regions associated with experiential knowledge. Importantly, at the neural level, our data highlight differences in so called type 2 clinical decision-making processes related to prior knowledge and experience.