Program directors' reflections on national policy change in medical education: insights on decision-making, accreditation, and the CanMEDS framework.

BACKGROUND: Outcomes of national policy change impact all levels of the organizational hierarchy. The medical education literature is sparse on how reflections from program directors (PDs) on past large-scale policy changes can inform future policy initiatives. To fill this gap, we conducted a national survey on PDs' perceptions of, and reflections on, decision-making in medical education, accreditation procedures, and the CanMEDS framework implementation. METHODS: The survey was distributed to former Canadian specialty medicine PDs (N = 684). Descriptive analysis was performed on quantitative data, thematic analysis was performed on qualitative comments, and comparisons between the quantitative and qualitative findings were performed to identify areas of convergence and/or divergence. RESULTS: A total of 265 (38.7%) former PDs participated. Quantitative analysis revealed that 52.8% of respondents did not feel involved in decision-making regarding policy changes, 45.1% of respondents did not feel prepared to assess the CanMEDS Roles, and PDs were divided on the reasonableness of accreditation documentation. Qualitative analysis produced four themes: communication, resources, expectations of outcomes, and buy-in. Nine sub-themes were also identified. A high level of convergence was identified across the content, with only four areas of divergence identified. CONCLUSIONS: Our findings have the potential to inform future policy and/or accreditation changes. Without the lens of those charged with overseeing the implementation, policy evaluation and quality improvement will remain uninformed. PDs, therefore, bring unique insights into our understanding of national policy changes, and without the voices of these frontline implementers, the true success of policy change implementation will be hindered.

CONTEXTE: Les effets des changements apportés aux politiques nationales se font sentir à tous les niveaux de la hiérarchie organisationnelle. La littérature traite peu du fait que l'opinion des directeurs de programme (DP) concernant les réformes d'envergure intervenues dans les politiques sur l'éducation médicale par le passé peut servir à éclairer les révisions de politiques futures. Afin de combler cette lacune, nous avons mené une enquête nationale pour sonder les DP sur leurs perceptions et réflexions quant à la prise de décision dans l'éducation médicale, aux procédures d'agrément et à la mise en œuvre du cadre CanMEDS. MÉTHODES: Le sondage a été distribué aux anciens DP en médecine spécialisée du Canada (N = 684). Les données quantitatives ont fait l'objet d'une analyse descriptive, les commentaires qualitatifs d'une analyse thématique, et une comparaison entre les résultats quantitatifs et qualitatifs a été effectuée pour repérer les domaines de convergence et de divergence. RÉSULTATS: Un total de 265 (38.7%) anciens DP ont participé au sondage. L'analyse quantitative a révélé que 52.8% des répondants ne se sentaient pas inclus dans la prise de décision en matière de changements de politiques, que 45.1% des répondants ne se sentaient pas en mesure d'évaluer les rôles CanMEDS, et qu'ils étaient partagés sur la question du caractère raisonnable des documents d'agrément. L'analyse qualitative a permis de dégager quatre thèmes: la communication, les ressources, les attentes en matière de résultats et l'adhésion. Neuf sous-thèmes ont également été définis. Nous avons constaté un niveau élevé de convergence sur l'ensemble du contenu, des divergences n'apparaissant que dans quatre domaines. CONCLUSIONS: Nos conclusions peuvent servir à orienter les changements futurs en matière de politiques et d'agrément. Sans le regard de ceux qui sont chargés de superviser leur mise en œuvre, l'évaluation des politiques et l'amélioration de la qualité demeureront mal fondées. La perspective unique des DP est essentielle à notre compréhension des révisions des politiques, et sans la contribution de ces responsables de première ligne de leur application, les réformes ne pourront être mises en œuvre de façon optimale.

Addressing the Diversity-Validity Dilemma Using Situational Judgment Tests.

PURPOSE: To examine the magnitudes of score differences across different demographic groups for three academic (grade point average [GPA], old Medical College Admission Test [MCAT], and MCAT 2015) and one nonacademic (situational judgment test [SJT]) screening measures and one nonacademic (multiple mini-interview [MMI]) interview measure (analysis 1), and the demographic implications of including an SJT in the screening stage for the pool of applicants who are invited to interview (analysis 2). METHOD: The authors ran the analyses using data from New York Medical College School of Medicine applicants from the 2015-2016 admissions cycle. For analysis 1, effect sizes (Cohen d) were calculated for GPA, old MCAT, MCAT 2015, CASPer (an online SJT), and MMI. Comparisons were made across gender, race, ethnicity (African American, Hispanic/Latino), and socioeconomic status (SES). For analysis 2, a series of simulations were conducted to estimate the number of underrepresented in medicine (UIM) applicants who would have been invited to interview with different weightings of GPA, MCAT, and CASPer scores. RESULTS: A total of 9,096 applicants were included in analysis 1. Group differences were significantly smaller or reversed for CASPer and MMI compared with the academic assessments (MCAT, GPA) across nearly all demographic variables/indicators. The simulations suggested that a higher weighting of CASPer may help increase gender, racial, and ethnic diversity in the interview pool; results for low-SES applicants were mixed. CONCLUSIONS: The inclusion of an SJT in the admissions process has the potential to widen access to medical education for a number of UIM groups.

Assessing the Quality of Evidence Presented at Annual General Meetings: A 5-Year Retrospective Study.

INTRODUCTION: Health care professionals rely on annual general meetings (AGMs) to obtain up-to-date information and practice guidelines relevant to their specialty. The majority of such information at meetings is presented through abstract sessions. However, the quality of the evidence presented during such abstract sessions is unclear. Standardized measures were applied to assess the quality of evidence of abstracts presented at the Canadian Society of Nephrology AGM over a 5-year period. METHODS: Two authors independently reviewed all CSN AGM abstracts presented from 2012 to 2016. Using a schema published in 2011 by the Oxford Centre for Evidence-Based Medicine (OCEBM), each abstract was subsequently ranked based on the quality of evidence. Schema categories ranged from level I, representing the highest evidence quality, to level V, representing the lowest. The number of authors and the authors' institution affiliations were also collected from the abstracts, where available, or if affiliations were unclear, an internet search of the author was performed. RESULTS: Six hundred forty-two articles were screened. In total, 70% (n = 450) met the inclusion criteria. When assessed, 15% of articles were level I (highest quality), 17% level II, 53% level III, 12% level IV, and 3% level V (lowest quality). A Jonckheere-Terpstra test demonstrated a significant trend of increasing quality of evidence (P < .05) and collaboration (P < .005) over the 5-year study period. There was a significant correlation between level of evidence and collaboration across years reviewed in the study, rs(98) = -0.226, P < .001. DISCUSSION: The results indicate a consistent increase in quality of evidence and collaborative submissions over time. To the authors' knowledge, this is the first assessment and analysis of AGM presentation quality within internal medicine and its subspecialties. Documenting and monitoring changes in the quality of evidence with a standardized framework may offer valuable insight pertaining to the medical field and the research community.

Correction to: CASPer, an online pre-interview screen for personal/professional characteristics: prediction of national licensure scores.

In re-examining the paper "CASPer, an online pre-interview screen for personal/professional characteristics: prediction of national licensure scores" published in AHSE (22(2), 327-336), we recognized two errors of interpretation.

CASPer, an online pre-interview screen for personal/professional characteristics: prediction of national licensure scores.

Typically, only a minority of applicants to health professional training are invited to interview. However, pre-interview measures of cognitive skills predict for national licensure scores (Gauer et al. in Med Educ Online 21 2016) and subsequently licensure scores predict for performance in practice (Tamblyn et al. in JAMA 288(23): 3019-3026, 2002; Tamblyn et al. in JAMA 298(9):993-1001, 2007). Assessment of personal and professional characteristics, with the same psychometric rigour of measures of cognitive abilities, are needed upstream in the selection to health profession training programs. To fill that need, Computer-based Assessment for Sampling Personal characteristics (CASPer)-an on-line, video-based screening test-was created. In this paper, we examine the correlation between CASPer and Canadian national licensure examination outcomes in 109 doctors who took CASPer at the time of selection to medical school. Specifically, CASPer scores were correlated against performance on cognitive and 'non-cognitive' subsections of both the Medical Council of Canada Qualifying Examination (MCCQE) Parts I (end of medical school) and Part II (18 months into specialty training). Unlike most national licensure exams, MCCQE has specific subcomponents examining personal/professional qualities, providing a unique opportunity for comparison. The results demonstrated moderate predictive validity of CASPer to national licensure outcomes of personal/professional characteristics three to six years after admission to medical school. These types of disattenuated correlations (r = 0.3-0.5) are not otherwise predicted by traditional screening measures. These data support the ability of a computer-based strategy to screen applicants in a feasible, reliable test, which has now demonstrated predictive validity, lending evidence of its validation for medical school applicant selection.

Disrupting diagnostic reasoning: do interruptions, instructions, and experience affect the diagnostic accuracy and response time of residents and emergency physicians?

PURPOSE: Others have suggested that increased time pressure, sometimes caused by interruptions, may result in increased diagnostic errors. The authors previously found, however, that increased time pressure alone does not result in increased errors, but they did not test the effect of interruptions. It is unclear whether experience modulates the combined effects of time pressure and interruptions. This study investigated whether increased time pressure, interruptions, and experience level affect diagnostic accuracy and response time. METHOD: In October 2012, 152 residents were recruited at five Medical Council of Canada Qualifying Examination Part II test sites. Forty-six emergency physicians were recruited from one Canadian and one U.S. academic health center. Participants diagnosed 20 written general medicine cases. They were randomly assigned to receive fast (time pressure) or slow condition instructions. Visual and auditory case interruptions were manipulated as a within-subject factor. RESULTS: Diagnostic accuracy was not affected by interruptions or time pressure but was related to experience level: Emergency physicians were more accurate (71%) than residents (43%) (F = 234.0, P < .0001) and responded more quickly (54 seconds) than residents (65 seconds) (F = 9.0, P < .005). Response time was shorter for participants in the fast condition (55 seconds) than in the slow condition (73 seconds) (F = 22.2, P < .0001). Interruptions added about 8 seconds to response time. CONCLUSIONS: Experienced emergency physicians were both faster and more accurate than residents. Instructions to proceed quickly and interruptions had a small effect on response time but no effect on accuracy.

Calling for a broader conceptualization of diversity: surface and deep diversity in four Canadian medical schools.

PURPOSE: Policy groups recommend monitoring and supporting more diversity among medical students and the medical workforce. In Canada, few data are available regarding the diversity of medical students, which poses challenges for policy development and evaluation. The authors examine diversity through a framework of surface (visible) and deep (less visible) dimensions and present data regarding a sample of Canadian medical students. METHOD: Between 2009 and 2011, nine cohorts from four Canadian medical schools completed the Health Professions Student Diversity Survey (HPSDS) either on paper or online. Items asked each participant's age, gender, gender identity, sexual identity, marital status, ethnicity, rural status, parental income, and disability. Data were analyzed descriptively and compared, when available, with national data. RESULTS: Of 1,892 students invited, 1,552 (82.0%) completed the HPSDS. Students tended to be 21 to 25 years old (68.3%; 1,048/1,534), female (59.0%; 902/1,529), heterosexual (94.6%; 1,422/1,503), single (90.1%; 1,369/1,520), and unlikely to report any disability (96.5%; 1,463/1,516). The majority of students identified with the gender on their birth certificate (99.8%; 1,512/1,515). About half had spent the majority of their lives in urban environments (46.7%; 711/1,521), and most reported parental household incomes of over $100,000/year (57.6%; 791/1,373). Overall, they were overrepresentative of higher-income groups and underrepresentative of populations of Aboriginal, black, or Filipino ethnicities in Canada. CONCLUSIONS: The authors propose the development of a National Student Diversity Database to support both locally relevant policies regarding pipeline programs and an examination of current application and selection procedures to identify potential barriers for underrepresented students.

The relationship between response time and diagnostic accuracy.

PURPOSE: Psychologists theorize that cognitive reasoning involves two distinct processes: System 1, which is rapid, unconscious, and contextual, and System 2, which is slow, logical, and rational. According to the literature, diagnostic errors arise primarily from System 1 reasoning, and therefore they are associated with rapid diagnosis. This study tested whether accuracy is associated with shorter or longer times to diagnosis. METHOD: Immediately after the 2010 administration of the Medical Council of Canada Qualifying Examination (MCCQE) Part II at three test centers, the authors recruited participants, who read and diagnosed a series of 25 written cases of varying difficulty. The authors computed accuracy and response time (RT) for each case. RESULTS: Seventy-five Canadian medical graduates (of 95 potential participants) participated. The overall correlation between RT and accuracy was -0.54; accuracy, then, was strongly associated with more rapid RT. This negative relationship with RT held for 23 of 25 cases individually and overall when the authors controlled for participants' knowledge, as judged by their MCCQE Part I and II scores. For 19 of 25 cases, accuracy on each case was positively related to experience with that specific diagnosis. A participant's performance on the test overall was significantly correlated with his or her performance on both the MCCQE Part I and II. CONCLUSIONS: These results are inconsistent with clinical reasoning models that presume that System 1 reasoning is necessarily more error prone than System 2. These results suggest instead that rapid diagnosis is accurate and relates to other measures of competence.

Influence of familiar features on diagnosis: instantiated features in an applied setting.

Medical diagnosis can be viewed as a categorization task. There are two mechanisms whereby humans make categorical judgments: "analytical reasoning," based on explicit consideration of features and "nonanalytical reasoning," an unconscious holistic process of matching against prior exemplars. However, there is evidence that prior experience can also operate at the level of individual "instantiated" features (Brooks & Hannah, 2006). The present studies examined instantiated features in medical diagnosis. Four "pseudopsychiatric" conditions, each described by four characteristic features, were taught to undergraduate psychology students. They practiced on additional cases, then were tested on new cases with features from two conditions. In Experiment 1, diagnoses associated with familiar features presented one or three times during practice were assigned a higher probability than those with novel features. Experiment 2 showed that the impact of feature frequency was dependent on its consistency with the case diagnosis. Experiment 3 showed that the effect of feature familiarity was not confined to cases with two equiprobable diagnoses. Experiment 4 showed that the effect remained after a 24 hour delay. These four studies demonstrated that features seen in practice have a greater influence on diagnosis than novel synonyms. In fact, seeing a feature once within the appropriate context (a patient case in which it is a member of the primary diagnosis) was sufficient to form a diagnostic association equivalent to instantiations seen four times in a different context. The results of these studies have implications for theories of categorization and for teaching clinical reasoning.

The effectiveness of cognitive forcing strategies to decrease diagnostic error: an exploratory study.

BACKGROUND: Cognitive forcing strategies, a form of metacognition, have been advocated as a strategy to prevent diagnostic error. Increasingly, curricula are being implemented in medical training to address this error. Yet there is no experimental evidence that these curricula are effective. DESCRIPTION: This was an exploratory, prospective study using consecutive enrollment of 56 senior medical students during their emergency medicine rotation. Students received interactive, standardized cognitive forcing strategy training. EVALUATION: Using a cross-over design to assess transfer between similar (to instructional cases) and novel diagnostic cases, students were evaluated on 6 test cases. Forty-seven students were immediately tested and 9 were tested 2 weeks later. Data were analyzed using descriptive statistics and a McNemar chi-square test. CONCLUSIONS: This is the first study to explore the impact of cognitive forcing strategy training on diagnostic error. Our preliminary findings suggest that application and retention is poor. Further large studies are required to determine if transfer across diagnostic formats occurs.

The reliability and acceptability of the Multiple Mini-Interview as a selection instrument for postgraduate admissions.

BACKGROUND: The Multiple Mini-Interview (MMI) is useful in selecting undergraduate medical trainees. Postgraduate applicant pools have smaller numbers of more homogeneous candidates that must be actively recruited while being assessed. This paper reports on the MMI's use in assessing residency candidates. METHOD: Canadian and international medical graduates to three residency programs--obstetrics-gynecology and pediatrics (McMaster University) and internal medicine (University of Alberta)--underwent the MMI for residency selection (n = 484) in 2008 and 2009. Reliability was determined and candidates and interviewers completed an exit survey assessing acceptability. RESULTS: Overall reliability of the MMI was acceptable, ranging from 0.55 to 0.72. Using 10 stations would increase reliability to 0.64-0.79. Eighty-eight percent of candidates believed they could accurately portray themselves, while 90% of interviewers believed they could reasonably judge candidates' abilities. CONCLUSIONS: The MMI provides a reliable way to assess residency candidates that is acceptable to both candidates and assessors across a variety of programs.

Extending the interview to all medical school candidates--Computer-Based Multiple Sample Evaluation of Noncognitive Skills (CMSENS).

BACKGROUND: Most medical school candidates are excluded without benefit of noncognitive skills assessment. Is development of a noncognitive preinterview screening test that correlates with the well-validated Multiple Mini-Interview (MMI) possible? METHOD: Study 1: 110 medical school candidates completed MMI and Computer-based Multiple Sample Evaluation of Noncognitive Skills (CMSENS)-eight 1-minute video-based scenarios and four self-descriptive questions, with short-answer-response format. Seventy-eight responses were audiotaped, 32 typewritten; all were scored by two independent raters. Study 2: 167 candidates completed CMSENS-eight videos, six self-descriptive questions, typewritten responses only, scored by two raters; 88 of 167 underwent the MMI. RESULTS: Results for overall test generalizability, interrater reliability, and correlation with MMI, respectively, were, for Study 1, audio-responders: 0.86, 0.82, 0.15; typewritten-responders: 0.72, 0.81, 0.51; and for Study 2, 0.83, 0.95, 0.46 (correlation with disattenuation was 0.60). CONCLUSIONS: Strong psychometric properties, including MMI correlation, of CMSENS warrant investigation into future widespread implementation as a preinterview noncognitive screening test.

Medical school admissions: revisiting the veracity and independence of completion of an autobiographical screening tool.

BACKGROUND: Some form of candidate-written autobiographical submission (ABS) is commonly used before interviews to screen candidates to medical school on the basis of their noncognitive characteristics. However, confidence in the validity of these measures has been questioned. METHOD: In 2005, applicants to McMaster University completed an off-site ABS before being interviewed and an on-site ABS at interview. Five off-site ABS questions were completed, plus eight on-site questions. On-site ABS questions were answered in variable timing conditions. ABS ratings were compared across sites and time allowed for completion. RESULTS: Off-site ABS ratings were higher than on-site ratings, and the two sets of ratings were uncorrelated with one another. On-site ABS ratings increased with increased time allowed for completion, but the reliability of the measure was unaffected by this variable. CONCLUSIONS: Confidence that candidates independently answer preinterview ABS questions is weak. To improve ABS validity, modification of the current Web-based submission format warrants consideration.

Medical school admissions: enhancing the reliability and validity of an autobiographical screening tool.

BACKGROUND: Most medical school applicants are screened out preinterview. Some cognitive scores available preinterview and some noncognitive scores available at interview demonstrate reasonable reliability and predictive validity. A reliable preinterview noncognitive measure would relax dependence upon screening based entirely on cognitive tendencies. METHOD: In 2005, applicants interviewing at McMaster University's Michael G. DeGroote School of Medicine completed an offsite, noninvigilated, Autobiographical Submission (ABS) preinterview and another onsite, invigilated, ABS at interview. Traditional and new ABS scoring methods were compared, with raters either evaluating all ABS questions for each candidate in turn (vertical scoring-traditional method) or evaluating all candidates for each question in turn (horizontal scoring-new method). RESULTS: The new scoring method revealed lower internal consistency and higher interrater reliability relative to the traditional method. More importantly, the new scoring method correlated better with the Multiple Mini-Interview (MMI) relative to the traditional method. CONCLUSIONS: The new ABS scoring method revealed greater interrater reliability and predictive capacity, thus increasing its potential as a screen for noncognitive characteristics.