Patient Engagement in Medical Trainee Selection: A Scoping Review.

PURPOSE: The stakes of medical trainee selection are high, making it ironic and somewhat paradoxical that patients and the public often get little say in selection practices. The authors sought to undertake a knowledge synthesis to uncover what is known about patient engagement across the medical trainee selection continuum. METHOD: The authors conducted a scoping review aimed at exploring the current state of practice and research on patient engagement in medical trainee selection in 2017-2021. MeSH headings and keywords were used to capture patient, community, and standardized patient engagement in selection processes across multiple health professions. The authors employed broad inclusion criteria and iteratively refined the corpus, ultimately, limiting study selection to those reporting engagement of actual patients in selection within medicine, but maintaining a broad focus on any patient contributions across the entire selection continuum. The Cambridge Framework was adapted and used to organize the included studies. RESULTS: In total, 2,858 abstracts were reviewed, and ultimately, 28 papers were included in the final corpus. The included studies were global but nascent. Most of the literature on this topic appears in the form of individual projects advocating for patient engagement in selection rather than cohesive programs with empirical exploration of patient engagement in selection. Job analysis methodology was particularly prominent for incorporating the patient voice into identifying competencies of relevance to selection. Direct patient engagement in early selection activities allowed the patient voice to assist candidates in determining their fit for medicine. CONCLUSIONS: Patient engagement has not been made a specific focus of study in its own right, leading the authors to encourage researchers to turn their lens more directly on patient engagement to explore how it complements the professional voice in medical trainee selection.

Collaborating With Service Users to Select Psychiatry Residents Committed to Health Equity and Social Justice.

PROBLEM: Advocates have called for health services to be delivered equitably to all. Academic psychiatry must play a role in this work, given its history of creating and perpetuating the marginalization of people experiencing mental health issues. While medical educators have started teaching concepts such as structural competency and cultural safety, careful consideration of who enters the medical workforce and what values they bring is also important. APPROACH: The authors report on the first 5 years (2016-2021) of a collaboration with individuals who have used mental health or addiction services or identify as having lived experiences of mental health and/or substance use issues (i.e., service users) to select residents to the general adult psychiatry residency program at the University of Toronto who are committed to working toward health equity and social justice and who bring diverse personal, academic, and community-based experiences. Starting in 2016, a working group of service users and faculty iteratively refined the selection process to add personal letter and interview day writing sample prompts centered on social justice and advocacy. OUTCOMES: The working group, coled by service users since 2019, defined the problem (lack of attention to health equity and social justice in resident selection) and codesigned the solution by revising writing prompts used in the selection process and their assessment rubrics to emphasize these missing areas. Further, service users directly participated in the implementation by reviewing candidates' personal letters and interview day writing samples alongside faculty and residents. This work serves as an example of meaningful service user engagement in action. NEXT STEPS: To ensure the needs of service users are prioritized, future work must aim for long-term institutional commitment to strengthen service user involvement and power sharing with service user communities in resident selection and at other points along the medical education pathway.

Adapting the Admissions Interview During COVID-19: A Comparison of In-Person and Video-Based Interview Validity Evidence.

COVID-19 physical distancing limited many medical schools' abilities to conduct in-person interviews for the 2020 admissions cycle. The University of Toronto (U of T) Temerty Faculty of Medicine was already in the midst of its interview process, with two-thirds of applicants having completed the in-person modified personal interview (MPI). As the university and surrounding region were shut down, the shift was made in the middle of the application cycle to a semisynchronous video-based MPI interview (vMPI) approach. U of T undertook the development, deployment, and evaluation of the 2 approaches mid-admissions cycle. Existing resources and tools were used to create a tailored interview process with the assistance of applicants. The vMPI was similar in content and process to the MPI: a 4-station interview with each station mapped to attributes relevant to medical school success. Instead of live interviews, applicants recorded 5-minute responses to questions for each station using their own hardware. These responses were later assessed by raters asynchronously. Out of 627 applicants, 232 applicants completed the vMPI. Validity evidence was generated for the vMPI and compared with the MPI on the internal structure, relationship to other variables, and consequential validity, including applicant and interviewer acceptability. Overall, the vMPI demonstrated similar reliability and factor structure to the MPI. As with the MPI, applicant performance was predicted by nonacademic screening tools but not academic measures. Applicants' acceptance of the vMPI was positive. Most interviewers found the vMPI to be acceptable and reported confidence in their ratings. Continuing physical distancing concerns will require multiple options for admissions committees to select medical students. The vMPI is an example of a customized approach that schools can implement and may have advantages for selection beyond the COVID-19 pandemic. Future evaluation will examine additional validity evidence for the tool.

Case Report: Defining Applicant Attributes to Be Prioritized in the Selection of Child and Adolescent Psychiatry Subspecialty Residents at the University of Toronto.

Background/Objectives: The child and adolescent psychiatry (CAP) subspecialty training program at the University of Toronto was among the first fully accredited CAP programs in Canada. As one of Canada's largest CAP subspecialty programs, we attract many excellent applicants annually. While objectivity and transparency in the selection of candidates have been valued, it was unclear which applicant attributes should be prioritized. This quality improvement project was undertaken to identify the key applicant attributes that should be prioritized for admission to the program. Materials/Methods: An initial list of attributes was compiled by project team members and feedback solicited. Through iterative design, this list was categorized into "end products," "branding attributes" and "generic attributes." The "end products" were removed as these represented outputs of training rather than attributes on which applicant selection should be based. Subsequent steps involved only the "branding" and "generic" attributes. A consensus-building exercise led to the creation of two short-lists of five attributes within each category. Finally, a paired-comparison forced choice methodology was used to determine the ranking of these attributes in order of importance when assessing applicants. Results: The final lists of "generic" and "branding" attributes developed through a consensus-building exercise are presented in rank order based on the paired-comparison methodology. The overall response rate for the forced choice electronic survey was 49% of faculty and learners. Conclusions/Discussion: This project used an iterative process of consensus building & pairwise comparison to prioritize key attributes for assessing trainee selection to the program. Going forward, these attributes will be incorporated into the file review and interview portions of our admissions process. In addition to emphasizing these priority attributes in admissions, there are implications for other aspects of the program including curriculum and faculty development, as well as guiding the overall mission and vision for the Division. A similar process could be undertaken by other training programs seeking to identify priority attributes for admission to their programs.

Patients, Pride, and Prejudice: Exploring Black Ontarian Physicians' Experiences of Racism and Discrimination.

PURPOSE: Black physicians' and trainees' experiences of racism are not well documented in Canada, reflecting a knowledge gap needing correction to combat racism in Canadian health care. The authors undertook a descriptive study of Black physicians and trainees in the Canadian province of Ontario. The goal of this study was to report upon racism experienced by participant Ontarian physicians to challenge the purported rarity of racism in Canadian health care. METHOD: An anonymous online survey of physicians and trainees who self-identify as Black (African/Afro-Canadian/African American/Afro-Caribbean) was administered in March and April 2018 through the Black Physicians' Association of Ontario (BPAO) listserv. The survey was modeled on qualitative interview guides from American studies. Snowball sampling was employed whereby BPAO members forwarded the survey to eligible colleagues (non-BPAO members) to maximize responses. Survey data were analyzed and key themes described. RESULTS: Survey participants totalled 46, with a maximal response rate of 38%. Participants reported positive experiences of collegiality with Black colleagues and strong bonds with Black patients. Negative discrimination experiences included differential treatment and racism from peers, superiors, and patients. Participants reported race as a major factor in their selection of practice location, more so than selection of career. Participants also expressed a lack of mentorship, and there was a strong call for increased mentorship from mentors with similar ethno-racial backgrounds. CONCLUSIONS: This study challenges the notion that racism within Canadian health care is rare. Future systematic collection of information regarding Black physicians' and trainees' experiences of racism will be key in appreciating the prevalence and nature of these experiences.

Exploring the Socialization Experiences of Medical Students From Social Science and Humanities Backgrounds.

PURPOSE: To explore the structural, cultural, and interpersonal issues that may contribute to the inadvertent marginalization of medical students with social science and humanities (SSH) backgrounds. METHOD: Using the hidden curriculum as an analytic construct, the lead author interviewed 14 medical students with SSH backgrounds at the University of Toronto Faculty of Medicine from February to October 2015. The authors analyzed the interview transcripts for common themes around positive and negative cultural, structural, and interpersonal dimensions of the socialization process. RESULTS: Participants reported barriers to applying to medical school: needing to complete prerequisite courses and to do well on an exam geared toward those with a strong science background (the Medical College Admission Test) and lacking an application cohort. Some participants felt they were not ideal candidates for medical school. Participants appreciated how their SSH backgrounds and associated skill sets shaped both their perspectives on patient care and their developing professional identities. However, they perceived that others largely deemed their previous training as irrelevant, and they felt marginalized in medical school by peers, instructors, and the curriculum. These experiences led both to self-censorship, which enabled them to seem to conform to normative behaviors, and to the pursuit of reaffirming elective experiences. CONCLUSIONS: The existing hidden curriculum inadvertently marginalizes SSH medical students; their experiences likely reflect the socialization experiences of other students from underrepresented backgrounds. Curricular and institutional reforms are imperative to shift the hidden curriculum toward one of epistemological inclusion that better supports students from nontraditional backgrounds.

A Reflection Upon the Impact of Early 21st-Century Technological Innovations on Medical School Admissions.

The authors describe influences associated with the incorporation of modern technologies into medical school admissions processes. Their purpose is not to critique or support specific technologies but, rather, to prompt reflection on the evolution that is afoot. Technology is now integral to the administration of multiple admissions tools, including the Medical College Admission Test, situational judgment tests, and standardized video interviews. Consequently, today's admissions landscape is transforming into an online, globally interconnected marketplace for health professions admissions tools. Academic capitalism and distance-based technologies combine to enable global marketing and dissemination of admissions tests beyond the national jurisdictions in which they are designed. As predicted by disruptive business theory, they are becoming key drivers of transformative change. The seeds of technological disruption are present now rather than something to be wary of in the future. The authors reflect on this transformation and the need for tailoring test modifications to address issues of medical student diversity and social responsibility. They comment on the online assessment of applicants' personal competencies and the potential detriments if this method were to replace admissions methods involving human contact, thanks to the ease with which institutions can implement them without cost to themselves and without adequate consideration of measurement utility or contextual appropriateness. The authors advocate for socially responsible academic capitalism within this interconnected admissions marketplace: Attending to today's transformative challenges may inform how health professions education responds to tomorrow's admissions technologies and, in turn, how tomorrow's health professionals respond to their patients' needs.

Thinking of selection and widening access as complex and wicked problems.

OBJECTIVES: 'Wicked problems' are complex in nature, have innumerable causes associated with multiple social environments and actors with unpredictable behaviour and outcomes, and are difficult to define or even resolve. This paper considers why and how the frameworks of complexity theory and wicked problems can help medical educators consider selection and widening access (WA) to medicine through fresh eyes to guide future policy and practice. We illustrate how 'wickedity' can frame the key issues in this area, and then address steps that education stakeholders might take to respond to and act on these issues. METHODS: We used the 10 properties of a wicked problem to frame common issues in the broad field of selection and WA in medicine. We drew heavily on literature from different disciplines, particularly education, and, through debate and reflection, agreed on the applicability of the theory for illuminating and potentially addressing outstanding issues in selection and WA. RESULTS: Framing medical school selection using the 10 properties of wicked problems is a means of shifting thinking from erroneous 'simple' solutions to thinking more contextually and receptively. The wicked problem framework positions selection as a multi-causal, complex, dynamic, social problem and foregrounds stakeholders' views and context as being highly relevant in medical school selection. CONCLUSIONS: The wicked problem lens shifts thinking and action from seeking one elusive, objective truth to recognising the complexity of medical school selection, managing uncertainty, questioning and considering 'issues' associated with medical school selection more productively. Although there are criticisms of this framework, labelling medical selection as 'wicked' provides original insights and genuine reframing of the challenges of this important, and high profile, aspect of medical education. Doing so, in turn, opens the door to different responses than would be the case if selection and WA were simple and readily tamed.

Opening Editorial: Selection and Recruitment in Medical Education.

This article was migrated. The article was marked as recommended. There is over a century of research on selection and recruitment and the field has both developed and expanded significantly over this time. Previous research has tended to focus on reviewing the effectiveness of selection methods (academic records, references, personal statements, aptitude tests, personality assessments, situational judgement tests, and interviews), where good quality evidence is now emerging. Many challenges remain however, reflecting that selection and recruitment into medical education (both undergraduate and postgraduate) is a complex, multi-dimensional, dynamic phenomenon. For example, issues regarding diversity and fairness in selection have been researched over many years but there remains a huge gap between the research evidence and policy enactment in many parts of the globe. In this opening editorial for our special issue on selection and recruitment in medical education we encourage authors to consider six key question areas (amongst others), including: •how will technology (e.g. social media, big data, artificial intelligence, etc) influence selection research and practices in future?•should selection criteria be reviewed to include creativity, innovation, resilience and adaptability (beyond heavy reliance on prior academic attainment as the main criterion)?•is selection for medical education fair? How do we address issues regarding widening participation and diversity in practice?•to what extent do political, cultural and social factors influence selection philosophy and policies internationally?•what are the risks to effective selection (e.g. access to coaching, legal challenge of poor practices) and,•a new Ottawa consensus on selection and recruitment has been published - to what extent does this statement reflect your experiences of designing and implementing selection systems in your locality? In contributing to the debate, this special issue provides a platform for authors to present the latest research, empirical studies, systematic reviews, reflections, case studies and practical tips on current/future issues in selection and recruitment in medical education.

The Direct Economic and Opportunity Costs of the Medical College Admissions Test (MCAT) for Canadian Medical Students.

This article was migrated. The article was marked as recommended. Background The Association of Faculties of Medicine of Canada, Future of Medical Education in Canada report shared a collective vision to improve social accountability, including a review of admissions policies to enhance student diversity. This study explored if and how the Medical College Admissions Test (MCAT) might mediate the socioeconomic diversity of Canadian medical schools by quantifying the costs and other cost-related factors of preparing for the exam. Methods A 34-question anonymous and bilingual (English and French) online questionnaire was sent to the 2015 first-year cohort of Canadian medical students. Developed collaboratively, the survey content focused on MCAT preparation and completion activities, associated costs, and students' perceptions of MCAT costs. Findings The survey response rate was 32%. First-year medical students were more likely than the Canadian population to belong to high-income families (63% vs. 36%) and less likely to be from rural locations (4.5% vs. 19%). Use of MCAT preparation materials was reported by nearly every MCAT test-taker (95.3%): of those, 76.4% used free practice tests; 59.8% paid for practice tests; 45.1% registered for preparation courses; and 3.3% hired a private tutor. In terms of writing the MCAT, the total economic costs per respondent are estimated at $6,357 ($4,755-$7,958) and total direct costs per respondent are estimated at $2,970 ($1,882- $4,058). Opportunity costs represented the majority of economic costs, at $3,387 ($2,872 - $3,901), or 53.2%. MCAT preparation costs are estimated to be $2,372 ($1,373-$3,372), or 79.9% of total direct costs and 37.3% of economic costs. Most respondents agreed, 76%, that the MCAT posed a financial hardship. Conclusion The financial demands of preparing for and completing the MCAT quantified in this study highlight an admissions requirement that is likely contributing to the current student diversity challenges in Canadian medical schools. In the spirit of social accountability, perhaps it is time to prioritize equitable alternative for assessing applicants' academic readiness for medical school.

Facilitating admissions of diverse students: A six-point, evidence-informed framework for pipeline and program development.

INTRODUCTION: Several national level calls have encouraged reconsideration of diversity issues in medical education. Particular interest has been placed on admissions, as decisions made here shape the nature of the future physician workforce. Critical analysis of current practices paired with evidence-informed policies may counter some of the barriers impeding access for underrepresented groups. METHODS: We present a framework for diversity-related program development and evaluation grounded within a knowledge translation framework, and supported by the initiation of longitudinal collection of diversity-related data. We provide an illustrative case study for each component of the framework. Descriptive analyses are presented of pre/post intervention diversity metrics if applicable and available. RESULTS: The framework's focal points are: 1) data-driven identification of underrepresented groups, 2) pipeline development and targeted recruitment, 3) ensuring an inclusive process, 4) ensuring inclusive assessment, 5) ensuring inclusive selection, and 6) iterative use of diversity-related data. Case studies ranged from wording changes on admissions websites to the establishment of educational and administrative offices addressing needs of underrepresented populations. CONCLUSIONS: We propose that diversity-related data must be collected on a variety of markers, developed in partnership with stakeholders who are most likely to facilitate implementation of best practices and new policies. These data can facilitate the design, implementation, and evaluation of evidence-informed diversity initiatives and provide a structure for continued investigation into 'interventions' supporting diversity-related initiatives.

Multiple independent sampling within medical school admission interviewing: an "intermediate approach".

INTRODUCTION: Balancing reliability and resource limitations as well as recruitment activities during admission interviews is a challenge for many medical schools. The Modified Personal Interview (MPI) has been shown to have good psychometric properties while being resource efficient for specialized admission interviews. We describe implementation of an MPI adaptation integrating psychometric rigour alongside resourcing and recruitment goals for larger-scale medical school admission interviewing at the University of Toronto. METHODS: The MPI was implemented during the 2013-2014 admission cycle. The MPI uses multiple independent sampling by having applicants interviewed in a circuit of four brief semi-structured interviews. Recruitment is reflected in a longer MPI interviewing time to foster a 'human touch'. Psychometric evaluation includes generalizability studies to examine inter-interview reliability and other major sources of error variance. We evaluated MPI impact upon applicant recruitment yield and resourcing. RESULTS: MPI reliability is 0.56. MPI implementation maintained recruitment compared with previous year. MPI implementation required 160 interviewers for 600 applicants whereas for pre-MPI implementation 290 interviewers were required to interview 587 applicants. MPI score correlated with first year OSCE performance at 0.30 (p < 0.05). DISCUSSION: MPI reliability is measured at 0.56 alongside enhanced resource utilization and maintenance of recruitment yield. This 'intermediate approach' may enable broader institutional uptake of integrated multiple independent sampling-based admission interviewing within institution-specific resourcing and recruitment goals.

Is Canada Ready for Nationwide Collaboration on Medical School Admissions Practices and Policies?

The report by the Association of Faculties of Medicine of Canada (AFMC) entitled "The Future of Medical Education in Canada: A Collective Vision for MD Education" includes recommendations to enhance admissions processes and increase national collaboration. To achieve these goals, the AFMC conducted a nationwide environmental scan appraising medical schools' readiness for national collaboration and progress toward establishing "made-in-Canada" admissions processes. A critical narrative review of the academic and gray literature was conducted as part of this environmental scan. Four core admissions practice and policy domains were identified: (1) social accountability strategies, (2) standardized admissions testing, (3) interviewing procedures, and (4) application procedures.In this article, the authors summarize and discuss the findings of this narrative review with regard to the four domains. They provide documentation of historical and present-day admissions factors relevant to Canadian medical schools' readiness for nationwide collaboration and a descriptive analysis of the facilitators and barriers to establishing "made-in-Canada" admissions processes.All four domains had facilitators and barriers. One barrier, however, cut across multiple domains-medical schools' pursuit of prestige and its potential to conflict with the goals of the other domains. The authors recommend holding a national forum to debate these issues and to advance the AFMC's goals, a process that will not be straightforward. Yet, national collaboration holds promise for applicants, medical schools, and Canada's diverse population of patients, so efforts toward this end must continue.

Investigating Canadian medical school attrition metrics to inform socially accountable admissions planning.

OBJECTIVE: Attrition from Canadian medical degree programs was never described despite differences in admissions requirements at the 17 faculties of medicine. Knowledge on attrition metrics could help the faculties evaluate new avenues for addressing the Association of Faculties of Medicine's (AFMC) Future of Medical Education in Canada (FMEC MD) recommendation to enhance admissions practices with the goal to improve social accountability and student diversity. METHOD: AFMC databases were used to track medical degree completion of all Canadian M.D. students who enrolled between 2003 and 2007. Students were followed and assigned an M.D. completion status as of by July 1, 2013. Bivariate statistics were used to evaluate if demographic, admission and degree progression variables were associated with medical school attrition. RESULTS: Of 11,454 students enrolled in Canadian M.D. programs from 2003 to 2007, only 197 (1.7%) did not complete. Québec had significantly higher attrition than other jurisdictions with age, educational attainment at time of enrolment, MCAT completion and struggling academically associated with attrition. CONCLUSION: Attrition from Canadian MD programs is rare and associated with differences in admission requirements and possibly suggests an optimum life stage for medical studies. Improved knowledge of attrition-related factors may offer an additional level of evidence for improving the alignment between admissions policies and the social accountability objectives of medical schools.

Exploring social media and admissions decision-making - friends or foes?

BACKGROUND: Despite the ever-increasing use of social media (e.g., Facebook, Twitter) little is known about its use in medical school admissions. This qualitative study explores whether and how social media (SM) is used in undergraduate admissions in Canada, and the attitudes of admissions personnel towards such use. METHODS: Phone interviews were conducted with admissions deans and nominated admissions personnel. A qualitative descriptive analysis was performed using iterative coding and comparing, and grouping data into themes. RESULTS: Personnel from 15 of 17 Canadian medical schools participated. A sizeable proportion had, at some point, examined social media (SM) profiles to acquire information on applicants. Participants did not report using it explicitly to screen all applicants (primary use); however, several did admit to looking at SM to follow up on preliminary indications of misbehaviour (secondary use). Participants articulated concerns, such as validity and equity, about using SM in admissions. Despite no schools having existing policy, participants expressed openness to future use. CONCLUSIONS: While some of the 15 schools had used SM to acquire information on applicants, criteria for formulating judgments were obscure, and participants expressed significant apprehension, based on concerns for fairness and validity. Findings suggest participant ambivalence and ongoing risks associated with "hidden" selection practices.

Identifying and Promoting Best Practices in Residency Application and Selection in a Complex Academic Health Network.

Medical education institutions have a social mandate to produce a diverse physician workforce that meets the public's needs. Recent reports have framed the admission process outcome of undergraduate and postgraduate medical education (UGME and PGME) programs as a key determinant of the collective contributions graduating cohorts will make to society, creating a sense of urgency around the issue of who gets accepted. The need for evidence-informed residency application and selection processes is growing because of the increasing size and diversity of the applicant pool and the need for equity, fairness, social accountability, and health human resource planning. The selection literature, however, is dominated by a UGME focus and emphasizes determination of desirable qualities of future physicians and selection instrument reliability and validity. Gaps remain regarding PGME selection, particularly the creation of specialty-specific selection criteria, suitable outcome measures, and reliable selection systems.In this Perspective, the authors describe the University of Toronto's centralized approach to defining system-level best practices for residency application and selection. Over the 2012-2013 academic year, the Best Practices in Application and Selection working group reviewed relevant literature and reports, consulted content experts, surveyed local practices, and conducted iterative stakeholder consultations on draft recommendations. Strong agreement arose around the resulting 13 principles and 24 best practices, which had either empirical support or face validity. These recommendations, which are shared in this article, have been adopted by the university's PGME advisory committee and will inform a national initiative to improve trainees' transition from UGME to PGME in Canada.