Sociotechnical imaginaries in academic medicine strategic planning: a document analysis.

Purpose Along with other industries, healthcare is becoming increasingly digitized. Our study explores how the field of academic medicine is preparing for this digital future. Method Active strategic plans available in English were collected from faculties of medicine in Canada (n = 14), departments in medical schools (n = 17), academic health science centres (n = 23) and associated research institutes (n = 5). In total, 59 strategic plans were subjected to a practice-oriented form of document analysis, informed by the concept of sociotechnical imaginaries. Results On the one hand, digital health is discursively treated as a continuation of the academic medicine vision, with expansions of physician competencies and of research institutes contributions. These imaginaries do not necessarily disrupt the field of academic medicine as currently configured. On the other hand, there is a vision of digital health pursuing a robust sociotechnical future with transformative implications for how care is conducted, what forms of knowledge are prioritized, how patients and patienthood will be understood, and how data work will be distributed. This imaginary may destabilize existing distributions of knowledge and power. Conclusions Looking through the lens of sociotechnical imaginaries, this study illuminates strategic plans as framing desirable futures, directing attention towards specific ways of understanding problems of healthcare, and mobilizing the resources to knit together social and technical systems in ways that bring these visions to fruition. There are bound to be tensions as these sociotechnical imaginaries are translated into material realities. Many of those tensions and their attempted resolutions will have direct implications for the expectations of health professional graduates, the nature of clinical learning environments, and future relationships with patients. Sociology of digital health and science and technology studies can provide useful insights to guide leaders in academic medicine shaping these digital futures.

Data sharing and big data in health professions education: Ottawa consensus statement and recommendations for scholarship.

Changes in digital technology, increasing volume of data collection, and advances in methods have the potential to unleash the value of big data generated through the education of health professionals. Coupled with this potential are legitimate concerns about how data can be used or misused in ways that limit autonomy, equity, or harm stakeholders. This consensus statement is intended to address these issues by foregrounding the ethical imperatives for engaging with big data as well as the potential risks and challenges. Recognizing the wide and ever evolving scope of big data scholarship, we focus on foundational issues for framing and engaging in research. We ground our recommendations in the context of big data created through data sharing across and within the stages of the continuum of the education and training of health professionals. Ultimately, the goal of this statement is to support a culture of trust and quality for big data research to deliver on its promises for health professions education (HPE) and the health of society. Based on expert consensus and review of the literature, we report 19 recommendations in (1) framing scholarship and research through research, (2) considering unique ethical practices, (3) governance of data sharing collaborations that engage stakeholders, (4) data sharing processes best practices, (5) the importance of knowledge translation, and (6) advancing the quality of scholarship through multidisciplinary collaboration. The recommendations were modified and refined based on feedback from the 2022 Ottawa Conference attendees and subsequent public engagement. Adoption of these recommendations can help HPE scholars share data ethically and engage in high impact big data scholarship, which in turn can help the field meet the ultimate goal: high-quality education that leads to high-quality healthcare.

Six ways to maximize survey response rates: lessons from a medical school accreditation survey in a Canadian setting.

Background: Surveys are being increasingly used to gather feedback and study data in healthcare professions. However, it may be challenging to achieve high response rates in surveys administered to healthcare professionals. The aim of this paper is to report six strategies that contributed to a high response rate on the Independent Student Analysis at the University of Toronto (U of T), which can be applied to other surveys to achieve strong response rates amongst healthcare professionals. Methods: In 2019, as part of accreditation for the U of T MD Program, we conducted the Independent Student Analysis, a student-led survey examining a medical student's experience. We review and critically evaluate the factors that contributed to a robust response rate amongst one of the largest cohorts of medical students in Canada. Results: Among 1080 students in the MD program, we achieved an unprecedented response rate of 87.2%. Six factors were identified that most contributed to our high response rate, including: faculty support, student representation, eliciting participant feedback, creating protected time for completion, offering incentives, and generating awareness. Conclusions: Eliciting high survey response rates from medical learners can be challenging. However, with careful consideration of learner feedback and effective employment of the strategies discussed in this paper, medical school faculty may better engage students in survey completion, achieving higher response rates and gathering richer insight, which can be used to more effectively enact meaningful change amongst healthcare professionals.

Contexte: Les enquêtes auprès de professionnels de la santé sont de plus en plus utilisées pour recueillir et étudier des perspectives et des données, mais il peut s'avérer difficile d'obtenir des taux de réponse élevés. Cet article vise à présenter six stratégies qui ont permis de susciter une forte participation à l'enquête de l'Analyse indépendante des étudiants à l'Université de Toronto et qui peuvent être transposées à d'autres contextes de sondage auprès des professionnels du domaine.En 2019, dans le cadre du processus d'agrément du programme de doctorat en médecine de l'Université de Toronto, nous avons réalisé l'Analyse indépendante des étudiants, une enquête dirigée par les étudiants en médecine visant à examiner leur expérience. Nous passons en revue et évaluons de manière critique les facteurs qui ont contribué à l'atteinte d'un taux de réponse élevé auprès d'une des plus grandes cohortes d'étudiants en médecine au Canada. Résultats: Nous avons obtenu un taux de réponse sans précédent de 87,2 % parmi les 1 080 étudiants inscrits au programme de médecine. Les six facteurs qui ont le plus contribué à ce taux de réponse élevé sont : le soutien du corps professoral, la représentation des étudiants, leur participation à la conception de l'enquête, la création de plages horaires réservées pour remplir le questionnaire, l'offre d'incitatifs et la sensibilisation. Conclusions: Il peut être difficile de susciter une forte participation aux questionnaires chez les étudiants en médecine. Cependant, en considérant attentivement les commentaires des apprenants et en utilisant efficacement les stratégies présentées dans cet article, le corps professoral seront en mesure de mobiliser les étudiants à participer davantage aux enquêtes et de tirer profit de leur apport pour éclairer la promotion de changements pertinents chez les professionnels de la santé.

Admitting privileges: A construction ecology perspective on the unintended consequences of medical school admissions.

Medical-school applicants learn from many sources that they must stand out to fit in. Many construct self-presentations intended to appeal to medical-school admissions committees from the raw materials of work and volunteer experiences, in order to demonstrate that they will succeed in a demanding profession to which access is tightly controlled. Borrowing from the field of architecture the lens of construction ecology, which considers buildings in relation to the global effects of the resources required for their construction, we reframe medical-school admissions as a social phenomenon that has far-reaching harmful unintended consequences, not just for medicine but for the broader world. Illustrating with discussion of three common pathways to experiences that applicants widely believe will help them gain admission, we describe how the construction ecology of medical school admissions can recast privilege as merit, reinforce colonizing narratives, and lead to exploitation of people who are already disadvantaged.

Artificial Intelligence in Undergraduate Medical Education: A Scoping Review.

PURPOSE: Artificial intelligence (AI) is a rapidly growing phenomenon poised to instigate large-scale changes in medicine. However, medical education has not kept pace with the rapid advancements of AI. Despite several calls to action, the adoption of teaching on AI in undergraduate medical education (UME) has been limited. This scoping review aims to identify gaps and key themes in the peer-reviewed literature on AI training in UME. METHOD: The scoping review was informed by Arksey and O'Malley's methodology. Seven electronic databases including MEDLINE and EMBASE were searched for articles discussing the inclusion of AI in UME between January 2000 and July 2020. A total of 4,299 articles were independently screened by 3 co-investigators and 22 full-text articles were included. Data were extracted using a standardized checklist. Themes were identified using iterative thematic analysis. RESULTS: The literature addressed: (1) a need for an AI curriculum in UME, (2) recommendations for AI curricular content including machine learning literacy and AI ethics, (3) suggestions for curriculum delivery, (4) an emphasis on cultivating "uniquely human skills" such as empathy in response to AI-driven changes, and (5) challenges with introducing an AI curriculum in UME. However, there was considerable heterogeneity and poor consensus across studies regarding AI curricular content and delivery. CONCLUSIONS: Despite the large volume of literature, there is little consensus on what and how to teach AI in UME. Further research is needed to address these discrepancies and create a standardized framework of competencies that can facilitate greater adoption and implementation of a standardized AI curriculum in UME.

Relational professional identity: How do pharmacy students see themselves in relation to others?

BACKGROUND: As the pharmacy profession moves towards patient-centred care, pharmacy schools have updated their curricula to prepare students for a full scope of practice. A critical objective of the new curricula is the professional socialization of pharmacy students into relational aspects of the profession: how pharmacists should interact with patients and other health care professionals. Through an examination of how one cohort of pharmacy students perceives its relationship to patients and physicians, this study aims to determine how these relational aspects of professional identity evolve with time spent in the program. METHODS: At 3 time points over a 2-year period, pharmacy students were asked to detail in writing how they would communicate with a physician concerning a hypothetical drug allergy scenario. A directed content analysis of their responses was conducted based on 3 main analytic categories: patient-centredness, physician collaboration and physician deference. These categories were further divided into 6 subcategories that were used as the variables for analysis. Statistical analyses examined longitudinal group trends for these variables. RESULTS: Over the 2 years of observation, an examination of the proportion of messages demonstrating the subcategories of interest showed that the only measure of the pharmacy students' relational professional identity that changed significantly over time occurred for the perception of a sense of shared care for the patient. All other aspects of their relational identity were stagnant and did not change as they progressed through training (χ2; 12.772, df = 2, p < 0.002). CONCLUSION: Our results suggest that the relational professional identity of participants was poorly developed with regards to both patients and physicians. Pharmacy educators must reexamine the methods currently being employed to foster students' professional identity development to ensure that new graduates are prepared to meet the challenges of a changing scope of practice. Can Pharm J (Ott) 2021;154:xx-xx.

Fostering trust, collaboration, and a culture of continuous quality improvement: A call for transparency in medical school accreditation.

Medical schools provide the foundation for a physician's growth and lifelong learning. They also require a large share of government resources. As such, they should seek opportunities to maintain trust from the public, their students, faculty, universities, regulatory colleges, and each other. The accreditation of medical schools attempts to assure stakeholders that the educational process conforms to appropriate standards and thus can be trusted. However, accreditation processes are poorly understood and the basis for accrediting authorities' decisions are often opaque. We propose that increasing transparency in accreditation could enhance trust in the institutions that produce society's physicians. While public reporting of accreditation results has been established in other jurisdictions, such as Australia and the United Kingdom, North American accrediting bodies have not yet embraced this more transparent approach. Public reporting can enhance public trust and engagement, hold medical schools accountable for continuous quality improvement, and can catalyze a culture of collaboration within the broader medical education ecosystem. Inviting patients and the public to peer into one of the most formative and fundamental parts of their physicians' professional training is a powerful tool for stakeholder and public engagement that the North American medical education community at large has yet to use.

Les facultés de médecine procurent les bases pour la croissance professionnelle et le développement professionnel continu. Elles absorbent également une grande part des ressources gouvernementales. Conséquemment, elles devraient chercher des occasions de maintenir la confiance du public, de leurs étudiants, du corps professoral, des universités, des organismes de réglementation et les unes des autres. L'accréditation des facultés de médecine vise à assurer les parties prenantes que le processus éducationnel est conforme aux normes appropriées et donc de confiance. Toutefois, les processus d'accréditation sont mal compris et les fondements des décisions d'accréditation des autorités sont souvent opaques.Nous proposons que l'accroissement de la transparence du processus d'agrément puisse rehausser la confiance dans les institutions qui forment les médecins de notre société. Bien que la diffusion publique des résultats de l'agrémentsoit établie dans d'autres juridictions, comme en Australie et au Royaume-Uni, les organismes d'agrément de l'Amérique du Nord n'ont pas encore adopté cette approche plus transparente. Les la diffusion publique peut améliorer la confiance et la participation du public, tenir les facultés responsables de l'amélioration continue de la qualité et catalyser une culture de collaboration au sein de l'écosystème élargi de la formation médicale. Inviter les patients et le public à scruter l'une des étapes les plus formative fondamentale de la formation professionnelle de leurs médecins est un puissant outil pour les parties prenantes ainsi que pour susciter la participation du public. Il reste à l'utiliser dans la communauté d'éducation médicale nord-américaine.

Same but Different: Exploring Mechanisms of Learning in a Longitudinal Integrated Clerkship.

PURPOSE: Longitudinal integrated clerkships (LICs) are a widely used method of delivering clerkship curricula. Although there is evidence that LICs work and core components of LIC training have been identified, there is insufficient understanding of which components are integral to why they work. To address this question, this research explored how students experienced the first year of an LIC program. The aim was to use participants' understanding of their learning experiences to identify potential mechanisms of the LIC curriculum model. METHOD: Thirty-two interviews were conducted with 13 University of Toronto students, 7 LIC and 6 block rotation students from the same site, from October 2014 to September 2015. A thematic analysis was performed iteratively to explore participants' understanding of their key learning experiences and outcomes. RESULTS: Participants in both cohorts described their key learning outcome as integration and application of knowledge during patient care. Experiences supporting this outcome were articulated as longitudinal variable practice and continuity of relationships with preceptors and patients. Critically, these experiences manifested differently for the 2 cohorts. For block students, these learning experiences appeared to reflect the informal curriculum, whereas for LIC students, learning experiences were better supported by the LIC formal curriculum. CONCLUSIONS: The results illustrate the importance of learning experiences that support longitudinality and continuity. By also emphasizing variability and knowledge integration, they align with literature on expert development. Notably, many of the learning experiences identified resulted from informal learning and thus support going beyond the formal curriculum when evaluating the effectiveness of curricula.

A Call to Investigate the Relationship Between Education and Health Outcomes Using Big Data.

There exists an assumption that improving medical education will improve patient care. While seemingly logical, this premise has rarely been investigated. In this Invited Commentary, the authors propose the use of big data to test this assumption. The authors present a few example research studies linking education and patient care outcomes and argue that using big data may more easily facilitate the process needed to investigate this assumption. The authors also propose that collaboration is needed to link educational and health care data. They then introduce a grassroots initiative, inclusive of universities in one Canadian province and national licensing organizations that are working together to collect, organize, link, and analyze big data to study the relationship between pedagogical approaches to medical training and patient care outcomes. While the authors acknowledge the possible challenges and issues associated with harnessing big data, they believe that the benefits supersede these. There is a need for medical education research to go beyond the outcomes of training to study practice and clinical outcomes as well. Without a coordinated effort to harness big data, policy makers, regulators, medical educators, and researchers are left with sometimes costly guesses and assumptions about what works and what does not. As the social, time, and financial investments in medical education continue to increase, it is imperative to understand the relationship between education and health outcomes.

Ultrasound Curricula in Undergraduate Medical Education: A Scoping Review.

The clinical applications of point-of-care ultrasound (US) have expanded rapidly over the past decade. To promote early exposure to point-of-care US, there is widespread support for the integration of US curricula within undergraduate medical education. However, despite growing evidence and enthusiasm for point-of-care US education in undergraduate medical education, the curricular design and delivery across undergraduate medical education programs remain variable without widely adopted national standards and guidelines. This article highlights the educational and teaching applications of point-of-care US with a focus on outcomes. We then review the evidence on curricular design, delivery, and integration and the assessment of competency for point-of-care US in undergraduate medical education.

The Direct Economic and Opportunity Costs of the Medical College Admissions Test (MCAT) for Canadian Medical Students.

This article was migrated. The article was marked as recommended. Background The Association of Faculties of Medicine of Canada, Future of Medical Education in Canada report shared a collective vision to improve social accountability, including a review of admissions policies to enhance student diversity. This study explored if and how the Medical College Admissions Test (MCAT) might mediate the socioeconomic diversity of Canadian medical schools by quantifying the costs and other cost-related factors of preparing for the exam. Methods A 34-question anonymous and bilingual (English and French) online questionnaire was sent to the 2015 first-year cohort of Canadian medical students. Developed collaboratively, the survey content focused on MCAT preparation and completion activities, associated costs, and students' perceptions of MCAT costs. Findings The survey response rate was 32%. First-year medical students were more likely than the Canadian population to belong to high-income families (63% vs. 36%) and less likely to be from rural locations (4.5% vs. 19%). Use of MCAT preparation materials was reported by nearly every MCAT test-taker (95.3%): of those, 76.4% used free practice tests; 59.8% paid for practice tests; 45.1% registered for preparation courses; and 3.3% hired a private tutor. In terms of writing the MCAT, the total economic costs per respondent are estimated at $6,357 ($4,755-$7,958) and total direct costs per respondent are estimated at $2,970 ($1,882- $4,058). Opportunity costs represented the majority of economic costs, at $3,387 ($2,872 - $3,901), or 53.2%. MCAT preparation costs are estimated to be $2,372 ($1,373-$3,372), or 79.9% of total direct costs and 37.3% of economic costs. Most respondents agreed, 76%, that the MCAT posed a financial hardship. Conclusion The financial demands of preparing for and completing the MCAT quantified in this study highlight an admissions requirement that is likely contributing to the current student diversity challenges in Canadian medical schools. In the spirit of social accountability, perhaps it is time to prioritize equitable alternative for assessing applicants' academic readiness for medical school.

Is Canada Ready for Nationwide Collaboration on Medical School Admissions Practices and Policies?

The report by the Association of Faculties of Medicine of Canada (AFMC) entitled "The Future of Medical Education in Canada: A Collective Vision for MD Education" includes recommendations to enhance admissions processes and increase national collaboration. To achieve these goals, the AFMC conducted a nationwide environmental scan appraising medical schools' readiness for national collaboration and progress toward establishing "made-in-Canada" admissions processes. A critical narrative review of the academic and gray literature was conducted as part of this environmental scan. Four core admissions practice and policy domains were identified: (1) social accountability strategies, (2) standardized admissions testing, (3) interviewing procedures, and (4) application procedures.In this article, the authors summarize and discuss the findings of this narrative review with regard to the four domains. They provide documentation of historical and present-day admissions factors relevant to Canadian medical schools' readiness for nationwide collaboration and a descriptive analysis of the facilitators and barriers to establishing "made-in-Canada" admissions processes.All four domains had facilitators and barriers. One barrier, however, cut across multiple domains-medical schools' pursuit of prestige and its potential to conflict with the goals of the other domains. The authors recommend holding a national forum to debate these issues and to advance the AFMC's goals, a process that will not be straightforward. Yet, national collaboration holds promise for applicants, medical schools, and Canada's diverse population of patients, so efforts toward this end must continue.

Use of an Objective Structured Assessment of Technical Skill After a Sports Medicine Rotation.

PURPOSE: The purpose of this study was to determine if the use of an Objective Structured Assessment of Technical skill (OSATS), using dry models, would be a valid method of assessing residents' ability to perform sports medicine procedures after training in a competency-based model. METHODS: Over 18 months, 27 residents (19 junior [postgraduate year (PGY) 1-3] and 8 senior [PGY 4-5]) sat the OSATS after their rotation, in addition to 14 sports medicine staff and fellows. Each resident was provided a list of 10 procedures in which they were expected to show competence. At the end of the rotation, each resident undertook an OSATS composed of 6 stations sampled from the 10 procedures using dry models-faculty used the Arthroscopic Surgical Skill Evaluation Tool (ASSET), task-specific checklists, as well as an overall 5-point global rating scale (GRS) to score each resident. Each procedure was videotaped for blinded review. RESULTS: The overall reliability of the OSATS (0.9) and the inter-rater reliability (0.9) were both high. A significant difference by year in training was seen for the overall GRS, the total ASSET score, and the total checklist score, as well as for each technical procedure (P < .001). Further analysis revealed a significant difference in the total ASSET score between junior (mean 18.4, 95% confidence interval [CI] 16.8 to 19.9) and senior residents (24.2, 95% CI 22.7 to 25.6), senior residents and fellows (30.1, 95% CI 28.2 to 31.9), as well as between fellows and faculty (37, 95% CI 36.1 to 27.8) (P < .05). CONCLUSIONS: The results of this study show that an OSATS using dry models shows evidence of validity when used to assess performance of technical procedures after a sports medicine rotation. However, junior residents were not able to perform as well as senior residents, suggesting that overall surgical experience is as important as intensive teaching. CLINICAL RELEVANCE: As postgraduate medical training shifts to a competency-based model, methods of assessing performance of technical procedures become necessary.

Evaluating the effect of distractions in the operating room on clinical decision-making and patient safety.

BACKGROUND: Answering telephone calls and pagers is common distraction in the operating room. We sought to evaluate the impact of distractions on patient care by (1) assessing the accuracy and safety of responses to clinical questions posed to a surgeon while operating and (2) determining whether pager distractions affect simulation-based surgical performance. METHODS: We conducted a randomized crossover study of obstetrics and gynecology residents. After studying a patient sign-out list, subjects performed a virtual salpingectomy. They were randomized to a distraction phase followed by quiet phase or vice versa. In the distraction phase, a pager beeped and subjects were asked questions based on the sign-out list. Accuracy of responses and the number of unsafe responses were recorded. In the quiet phase, trainees performed the task uninterrupted. Measures of surgical performance were successful task completion, time to task completion and operative blood loss. RESULTS: The mean score for correct responses to clinical questions during the distracted phase was 80 % (SD ±14 %). Nineteen residents (63 %) made at least 1 unsafe clinical decision while operating on the simulator (range 0-3). Subjects were more likely to successfully complete the surgical task in the allotted time under the quiet compared to distraction condition (OR 11.3, p = 0.03). There was no difference between the conditions in paired analysis for mean time (seconds) to task completion [426 (SD 133) vs. 440 (SD 186), p = 0.61] and mean operative blood loss (mL) [73.14 (SD 106) vs. 112.70 (SD 358), p = 0.47]. CONCLUSIONS: Distractions in the operating room may have a profound impact on patient safety on the wards. While multitasking in a simulated setting, the majority of residents made at least one unsafe clinical decision. Pager distractions also hindered surgical residents' ability to complete a simulated laparoscopic task in the allotted time without affecting other variables of surgical performance.

How to set the bar in competency-based medical education: standard setting after an Objective Structured Clinical Examination (OSCE).

BACKGROUND: The goal of the Objective Structured Clinical Examination (OSCE) in Competency-based Medical Education (CBME) is to establish a minimal level of competence. The purpose of this study was to 1) to determine the credibility and acceptability of the modified Angoff method of standard setting in the setting of CBME, using the Borderline Group (BG) method and the Borderline Regression (BLR) method as a reference standard; 2) to determine if it is feasible to set different standards for junior and senior residents, and 3) to determine the desired characteristics of the judges applying the modified Angoff method. METHODS: The results of a previous OSCE study (21 junior residents, 18 senior residents, and six fellows) were used. Three groups of judges performed the modified Angoff method for both junior and senior residents: 1) sports medicine surgeons, 2) non-sports medicine orthopedic surgeons, and 3) sports fellows. Judges defined a borderline resident as a resident performing at a level between competent and a novice at each station. For each checklist item, the judges answered yes or no for "will the borderline/advanced beginner examinee respond correctly to this item?" The pass mark was calculated by averaging the scores. This pass mark was compared to that created using both the BG and the BLR methods. RESULTS: A paired t-test showed that all examiner groups expected senior residents to get significantly higher percentage of checklist items correct compared to junior residents (all stations p < 0.001). There were no significant differences due to judge type. For senior residents, there were no significant differences between the cut scores determined by the modified Angoff method and the BG/BLR method. For junior residents, the cut scores determined by the modified Angoff method were lower than the cut scores determined by the BG/BLR Method (all p < 0.01). CONCLUSION: The results of this study show that the modified Angoff method is an acceptable method of setting different pass marks for senior and junior residents. The use of this method enables both senior and junior residents to sit the same OSCE, preferable in the regular assessment environment of CBME.

Quality of randomised controlled trials in medical education reported between 2012 and 2013: a systematic review protocol.

INTRODUCTION: Research in medical education has increased in volume over the past decades but concerns have been raised regarding the quality of trials conducted within this field. Randomised controlled trials (RCTs) involving educational interventions that are reported in biomedical journals have been criticised for their insufficient conceptual, theoretical framework. RCTs published in journals dedicated to medical education, on the other hand, have been questioned regarding their methodological rigour. The aim of this study is therefore to assess the quality of RCTs of educational interventions reported in 2012 and 2013 in journals dedicated to medical education compared to biomedical journals with respect to objective quality criteria. METHODS AND ANALYSIS: RCTs published between 1 January 2012 and 31 December 2013 in English are included. The search strategy is developed with the help of experienced librarians to search online databases for key terms. All of the identified RCTs are screened based on their titles and abstracts individually by the authors and then compared in pairs to assess agreement. Data are extracted from the included RCTs by independently scoring each RCT using a data collection form. The data collection form consists of four steps. Step 1 includes confirmation of RCT eligibility; step 2 consists of the CONSORT checklist; step 3 consists of the Medical Education Research Study Quality Instrument framework; step 4 consists of a Medical Education Extension (MEdEx) to the CONSORT checklist. The MEdEx includes the following elements: Description of scientific background, explanation of rationale, quality of research questions and hypotheses, clarity in the description of the use of the intervention and control as well as interpretation of results. ETHICS AND DISSEMINATION: This review is the first to systematically examine the quality of RCTs conducted in medical education. We plan to disseminate the results through publications and presentation at relevant conferences. Ethical approval is not sought for this review.

Cognition before curriculum: rethinking the integration of basic science and clinical learning.

PURPOSE: Integrating basic science and clinical concepts in the undergraduate medical curriculum is an important challenge for medical education. The health professions education literature includes a variety of educational strategies for integrating basic science and clinical concepts at multiple levels of the curriculum. To date, assessment of this literature has been limited. METHOD: In this critical narrative review, the authors analyzed literature published in the last 30 years (1982-2012) using a previously published integration framework. They included studies that documented approaches to integration at the level of programs, courses, or teaching sessions and that aimed to improve learning outcomes. The authors evaluated these studies for evidence of successful integration and to identify factors that contribute to integration. RESULTS: Several strategies at the program and course level are well described but poorly evaluated. Multiple factors contribute to successful learning, so identifying how interventions at these levels result in successful integration is difficult. Evidence from session-level interventions and experimental studies suggests that integration can be achieved if learning interventions attempt to link basic and clinical science in a causal relationship. These interventions attend to how learners connect different domains of knowledge and suggest that successful integration requires learners to build cognitive associations between basic and clinical science. CONCLUSIONS: One way of understanding the integration of basic and clinical science is as a cognitive activity occurring within learners. This perspective suggests that learner-centered, content-focused, and session-level-oriented strategies can achieve cognitive integration.

Admissions file review: applying the multiple independent sampling (MIS) methodology.

PURPOSE: Although multiple independent sampling (MIS) has been adapted for admissions interviews, its application for assessing written materials in the admissions file has been limited. Currently, admissions file review at the University of Toronto medical school involves one rater per file to enable holistic assessment, which may introduce a halo effect-that is, impressions of one component influencing the evaluation of other components. The authors examined whether MIS file review, through which multiple raters evaluate specific file components independently, may reduce this effect. METHOD: The authors selected a stratified random sample of 300 applicant files from the 2010-2011 admissions cycle for rescoring by MIS. They divided each of the 300 applicant files into their four components (academic transcript, autobiographical sketch, personal statement, reference letters) and rebundled them into packages of 38 same-component items (purposely creating some overlap among packages to assess inter-rater reliability). The authors distributed each package to 1 of 36 raters; thus, each rater evaluated only one of four components across many applicants. The authors compared the inter-component reliability and factor analysis of MIS with that of holistic scoring. RESULTS: Ratings were returned for all applicants. Inter-component reliability (Cronbach alpha) was 0.69 for holistic scoring and 0.29 for MIS. Factor analysis showed all components loading heavily onto one factor in the holistic approach and onto three factors in the MIS method. CONCLUSIONS: Using MIS to assess the admissions file may reduce the halo effect and should be considered when evaluating applicants' written submissions.

Modified personal interviews: resurrecting reliable personal interviews for admissions?

PURPOSE: Traditional admissions personal interviews provide flexible faculty-student interactions but are plagued by low inter-interview reliability. Axelson and Kreiter (2009) retrospectively showed that multiple independent sampling (MIS) may improve reliability of personal interviews; thus, the authors incorporated MIS into the admissions process for medical students applying to the University of Toronto's Leadership Education and Development Program (LEAD). They examined the reliability and resource demands of this modified personal interview (MPI) format. METHOD: In 2010-2011, LEAD candidates submitted written applications, which were used to screen for participation in the MPI process. Selected candidates completed four brief (10-12 minutes) independent MPIs each with a different interviewer. The authors blueprinted MPI questions to (i.e., aligned them with) leadership attributes, and interviewers assessed candidates' eligibility on a five-point Likert-type scale. The authors analyzed inter-interview reliability using the generalizability theory. RESULTS: Sixteen candidates submitted applications; 10 proceeded to the MPI stage. Reliability of the written application components was 0.75. The MPI process had overall inter-interview reliability of 0.79. Correlation between the written application and MPI scores was 0.49. A decision study showed acceptable reliability of 0.74 with only three MPIs scored using one global rating. Furthermore, a traditional admissions interview format would take 66% more time than the MPI format. CONCLUSIONS: The MPI format, used during the LEAD admissions process, achieved high reliability with minimal faculty resources. The MPI format's reliability and effective resource use were possible through MIS and employment of expert interviewers. MPIs may be useful for other admissions tasks.