Quantifying Novice and Expert Differences in Visual Diagnostic Reasoning in Veterinary Pathology Using Eye-Tracking Technology.

Visual diagnostic reasoning is the cognitive process by which pathologists reach a diagnosis based on visual stimuli (cytologic, histopathologic, or gross imagery). Currently, there is little to no literature examining visual reasoning in veterinary pathology. The objective of the study was to use eye tracking to establish baseline quantitative and qualitative differences between the visual reasoning processes of novice and expert veterinary pathologists viewing cytology specimens. Novice and expert participants were each shown 10 cytology images and asked to formulate a diagnosis while wearing eye-tracking equipment (10 slides) and while concurrently verbalizing their thought processes using the think-aloud protocol (5 slides). Compared to novices, experts demonstrated significantly higher diagnostic accuracy (p <.017), shorter time to diagnosis (p <.017), and a higher percentage of time spent viewing areas of diagnostic interest (p <.017). Experts elicited more key diagnostic features in the think-aloud protocol and had more efficient patterns of eye movement. These findings suggest that experts' fast time to diagnosis, efficient eye-movement patterns, and preference for viewing areas of interest supports system 1 (pattern-recognition) reasoning and script-inductive knowledge structures with system 2 (analytic) reasoning to verify their diagnosis.

Insight into team competence in medical, nursing and respiratory therapy students.

This study provides information for educators about levels of competence in teams comprised of medical, nursing and respiratory therapy students after receiving a simulation-based team-training (SBT) curriculum with and without an additional formalized 30-min team-training (TT) module. A two-group pre- and post-test research design was used to evaluate team competence with respect to leadership, roles and responsibilities, communication, situation awareness and resource utilization. All scenarios were digitally recorded and evaluated using the KidSIM Team Performance Scale by six experts from medicine, nursing and respiratory therapy. The lowest scores occurred for items that reflected situation awareness. All teams improved their aggregate scores from Time 1 to Time 2 (p < 0.05). Student teams in the intervention group achieved significantly higher performance scores at Time 1 (Cohen's d = 0.92, p < 0.001) and Time 2 (d = 0.61, p < 0.01). All student teams demonstrated significant improvement in their ability to work more effectively by Time 2. The results suggest that situational awareness is an advanced expectation for the undergraduate student team. The provision of a formalized TT module prior to engaging student teams in a simulation-based TT curriculum led to significantly higher performances at Time 1 and 2.

Educational interventions to improve cytology visual diagnostic reasoning measured by eye tracking.

The teaching of visual diagnostic reasoning skills, to date, has been conducted in a largely unstructured apprenticeship manner. The purpose of this study was to assess if the introduction of two educational interventions improved the visual diagnostic reasoning skills of novices. These were (1) the active use of key diagnostic features and (2) image repetition. A pre-test and post-test research design was used to compare the two teaching interventions to a traditional teaching group and an expert group using eye tracking as an assessment method. The time to diagnosis and the percentage of time spent viewing an area of diagnostic interest (AOI) were compared using independent t-tests, paired t-tests, and analysis of covariance (ANCOVA). Diagnostic accuracy as a dichotomous variable was compared using Chi-square tables. Students taught in an active-learning manner with image repetition behaved most like experts, with no significant difference from experts for percentage of time spent in the AOIs and a significantly faster time to diagnosis than experts (p<.017). Our results from the educational interventions suggest a greater level of improvement in the eye tracking of students that were taught key diagnostic features in an active-learning forum and were shown multiple case examples.

Development of an orthopedic surgery trauma patient handover checklist.

BACKGROUND: In surgery, preoperative handover of surgical trauma patients is a process that must be made as safe as possible. We sought to determine vital clinical information to be transferred between patient care teams and to develop a standardized handover checklist. METHODS: We conducted standardized small-group interviews about trauma patient handover. Based on this information, we created a questionnaire to gather perspectives from all Canadian Orthopaedic Association (COA) members about which topics they felt would be most important on a handover checklist. We analyzed the responses to develop a standardized handover checklist. RESULTS: Of the 1106 COA members, 247 responded to the questionnaire. The top 7 topics felt to be most important for achieving patient safety in the handover were comorbidities, diagnosis, readiness for the operating room, stability, associated injuries, history/mechanism of injury and outstanding issues. The expert recommendations were to have handover completed the same way every day, all appropriate radiographs available, adequate time, all appropriate laboratory work and more time to spend with patients with more severe illness. CONCLUSION: Our main recommendations for safe handover are to use standardized checklists specific to the patient and site needs. We provide an example of a standardized checklist that should be used for preoperative handovers. To our knowledge, this is the first checklist for handover developed by a group of experts in orthopedic surgery, which is both manageable in length and simple to use.

CONTEXTE: En chirurgie, le transfert préopératoire des polytraumatisés nécessitant une chirurgie est un processus que l'on doit rendre aussi sécuritaire que possible. Nous avons voulu déterminer quels renseignements cliniques vitaux doivent être transmis aux équipes de soins et préparer une liste de vérification standardisée à cette fin. MÉTHODES: Nous avons réalisé des entrevues standardisées par petits groupes au sujet du transfert des polytraumatisés. À partir des renseignements recueillis, nous avons élaboré un questionnaire pour obtenir le point de vue de tous les membres de l'Association canadienne d'orthopédie (ACO) au sujet des éléments jugés les plus importants sur une liste de vérification en vue du transfert. Nous avons analysé les réponses pour dresser une liste de vérification standardisée. RÉSULTATS: Sur les 1106 membres de l'ACO, 247 ont répondu au questionnaire. Les 7 éléments jugés les plus importants pour assurer la sécurité des patients lors du transfert ont été : comorbidités, diagnostic, état de préparation pour le bloc opératoire, stabilité, blessures connexes, histoire et mécanisme du traumatisme et questions en suspens. Les recommandations des experts ont été les suivantes : que les transferts s'effectuent de la même façon chaque jour, qu'on obtienne toutes les radiographies appropriées disponibles, qu'on dispose de temps suffisant, qu'on obtienne toute les analyses de laboratoire appropriées et qu'on aie plus de temps à consacrer aux patients plus grièvement blessés. CONCLUSION: Nos principales recommandations pour un transfert sécuritaire sont d'utiliser des listes de vérification standardisées spécifiques aux besoins des patients et des sites. Nous fournissons un modèle type de liste de vérification pour les transferts qui devrait être utilisée pour le transfert en chirurgie des polytraumatisés. À notre connaissance, il s'agit de la première liste de vérification rapide et simple mise au point à cette fin par un groupe d'experts en chirurgie orthopédique.

A simulation-based intervention teaching seizure management to caregivers: A randomized controlled pilot study.

OBJECTIVES: To examine the effect of simulation-based seizure management teaching on improving caregiver competence and reported confidence with managing seizures. The authors hypothesized that simulation-based education would lead to a higher level of demonstrated competence and reported confidence in family members and caregivers. Simulation has not been previously studied in this context. METHODS: A two-group pre- and post-test experimental research design involving a total of 61 caregivers was used. The intervention was a simulation-based seizure curriculum delivered as a supplement to traditional seizure discharge teaching. Caregiver performance was analyzed using a seizure management checklist. Caregivers' perception of self-efficacy was captured using a self-efficacy questionnaire. RESULTS: Caregivers in the experimental group achieved significantly higher postintervention performance scores than caregivers in the control group in both premedication and postmedication seizure management (P<0.01). Additionally, they achieved significantly higher scores on the self-efficacy questionnaire including items reflecting confidence managing the seizure at home (P<0.05). CONCLUSION: Caregivers receiving the supplemental simulation-based curriculum achieved significantly higher levels of competence and reported confidence, supporting a positive relationship between simulation-based seizure discharge education, and caregiver competence and confidence in managing seizures. Simulation sessions provided insight into caregiver knowledge but, more importantly, insight into the caregiver's ability to apply knowledge under stressful conditions, allowing tailoring of curriculum to meet individual needs. These findings may have applications and relevance for management of other acute or chronic medical conditions.

OBJECTIFS: Examiner l'effet de l'enseignement de la prise en charge des convulsions par simulation pour améliorer les compétences des soignants et leur confiance déclarée à traiter les convulsions. Les auteurs postulent que l'enseignement par simulation accroîtrait les compétences démontrées et la confiance déclarée des membres de la famille et des soignants. La simulation n'a jamais été étudiée dans ce contexte. MÉTHODOLOGIE: Une méthodologie de recherche expérimentale en deux groupes avant et après le test a été privilégiée auprès de 61 soignants. L'intervention consistait en un cours par simulation sur les convulsions donné en plus de l'enseignement habituel sur les convulsions présenté au congé. Le rendement des soignants a été analysé au moyen d'une liste de vérification de la prise en charge des convulsions. La perception d'auto-efficacité du soignant a été saisie au moyen d'un questionnaire d'auto-efficacité. RÉSULTATS: Les soignants du groupe expérimental ont obtenu des indices de rendement considérablement plus élevés après l'intervention que ceux du groupe témoin, tant avant qu'après la prise en charge des convulsions par médication (P<0,01). De plus, ils ont obtenu des indices considérablement plus élevés au questionnaire d'auto-efficacité, y compris les questions reflétant la confiance à soigner les convulsions à domicile (P<0,05). CONCLUSION: Les soignants qui avaient eu un cours par simulation ont obtenu des taux de compétence et de confiance déclarée beaucoup plus élevés. Ces résultats corroborent la relation positive entre l'enseignement sur les convulsions par simulation au congé et la compétence et la confiance des soignants envers la prise en charge des convulsions. Les séances de simulation donnaient un aperçu des compétences des soignants, mais, surtout, de leur capacité à appliquer leurs connaissances dans des conditions stressantes, ce qui permet d'adapter le cours à leurs besoins. Ces observations peuvent être utiles et pertinentes pour la prise en charge d'autres maladies aiguës ou chroniques.

Using simulation to improve the cognitive and psychomotor skills of novice students in advanced laparoscopic surgery: a meta-analysis.

Advances in simulation technologies have enhanced the ability to introduce the teaching and learning of laparoscopic surgical skills to novice students. In this meta-analysis, a total of 18 randomized controlled studies were identified that specifically looked at training novices in comparison with a control group as it pertains to knowledge retention, time to completion and suturing and knotting skills. The combined random-effect sizes (ESs) showed that novice students who trained on laparoscopic simulators have considerably developed better laparoscopic suturing and knot tying skills (d = 1.96, p < 0.01), conducted fewer errors (d = 2.13, p < 0.01), retained more knowledge (d = 1.57, p < 0.01) than their respective control groups, and were significantly faster on time to completion (d = 1.98, p < 0.01). As illustrated in corresponding Forest plots, the majority of the primary study outcomes included in this meta-analysis show statistically significant support (p < 0.05) for the use of laparoscopic simulators for novice student training on both knowledge and advanced surgical skill development (28 of 35 outcomes, 80%). The findings of this meta-analysis support strongly the use of simulators for teaching laparoscopic surgery skills to novice students in surgical residency programs.

Feasibility and fidelity of practising surgical fixation on a virtual ulna bone.

BACKGROUND: Surgical simulators provide a safe environment to learn and practise psychomotor skills. A goal for these simulators is to achieve high levels of fidelity. The purpose of this study was to develop a reliable surgical simulator fidelity questionnaire and to assess whether a newly developed virtual haptic simulator for fixation of an ulna has comparable levels of fidelity as Sawbones. METHODS: Simulator fidelity questionnaires were developed. We performed a stratified randomized study with surgical trainees. They performed fixation of the ulna using a virtual simulator and Sawbones. They completed the fidelity questionnaires after each procedure. RESULTS: Twenty-two trainees participated in the study. The reliability of the fidelity questionnaire for each separate domain (environment, equipment, psychological) was Cronbach α greater than 0.70, except for virtual environment. The Sawbones had significantly higher levels of fidelity than the virtual simulator (p < 0.001) with a large effect size difference (Cohen d < 1.3). CONCLUSION: The newly developed fidelity questionnaire is a reliable tool that can potentially be used to determine the fidelity of other surgical simulators. Increasing the fidelity of this virtual simulator is required before its use as a training tool for surgical fixation. The virtual simulator brings with it the added benefits of repeated, independent safe use with immediate, objective feedback and the potential to alter the complexity of the skill.

CONTEXTE: Les simulateurs chirurgicaux offrent un environnement sécuritaire pour apprendre et pour exercer les habiletés psychomotrices. L'un des objectifs de ces simu - lateurs est de produire des degrés élevés de fidélité. Le but de cette étude était de mettre au point un questionnaire fiable sur la fidélité des simulateurs chirurgicaux et de vérifier si un nouveau simulateur virtuel, avec interface haptique, pour la fixation du cubitus présentait des taux de fidélité comparables à ceux du simulateur Sawbones. MÉTHODES: Des questionnaires sur la fidélité des simulateurs ont été préparés. Nous avons procédé à une étude randomisée stratifiée auprès de stagiaires en chirurgie qui ont effectué une fixation du cubitus à l'aide du simulateur virtuel et à l'aide du simulateur Sawbones. Ils ont répondu au questionnaire sur la fidélité après chaque intervention. RÉSULTANTS: Vingt-deux stagiaires ont participé à l'étude. La fiabilité du questionnaire sur la fidélité pour chaque domaine distinct (environnement, équipement, dimension psychologique) correspondait à un coefficient α Cronbach supérieur à 0,70, sauf pour ce qui est de l'environnement virtuel. Le simulateur Sawbones a présenté des taux de fidélité significativement plus élevés que le simulateur virtuel (p < 0,001), avec une différence importante au plan de la taille de l'effet (indice d de Cohen < 1,3). CONCLUSIONS: Le nouveau questionnaire sur la fidélité s'est révélé un outil fiable qui peut servir à déterminer le degré de fidélité d'autres simulateurs chirurgicaux. Il faudra améliorer la fidélité de ce simulateur virtuel avant de pouvoir l'utiliser comme outil de formation pour la fixation chirurgicale. Ce simulateur virtuel a l'avantage de permettre des utilisations sécuritaires répétées et indépendantes avec des résultats immédiats et objectifs et de modifier la complexité de l'habileté.

Optimizing biomedical science learning in a veterinary curriculum: a review.

As veterinary medical curricula evolve, the time dedicated to biomedical science teaching, as well as the role of biomedical science knowledge in veterinary education, has been scrutinized. Aside from being mandated by accrediting bodies, biomedical science knowledge plays an important role in developing clinical, diagnostic, and therapeutic reasoning skills in the application of clinical skills, in supporting evidence-based veterinary practice and life-long learning, and in advancing biomedical knowledge and comparative medicine. With an increasing volume and fast pace of change in biomedical knowledge, as well as increased demands on curricular time, there has been pressure to make biomedical science education efficient and relevant for veterinary medicine. This has lead to a shift in biomedical education from fact-based, teacher-centered and discipline-based teaching to applicable, student-centered, integrated teaching. This movement is supported by adult learning theories and is thought to enhance students' transference of biomedical science into their clinical practice. The importance of biomedical science in veterinary education and the theories of biomedical science learning will be discussed in this article. In addition, we will explore current advances in biomedical teaching methodologies that are aimed to maximize knowledge retention and application for clinical veterinary training and practice.

The effect of simulation in improving students' performance in laparoscopic surgery: a meta-analysis.

BACKGROUND: There is increasing interest in using simulators for laparoscopic surgery training, and simulators have rapidly become an integral part of surgical education. METHODS: We searched MEDLINE, EMBASE, Cochrane Library, and Google Scholar for randomized controlled studies that compared the use of different types of simulators. The inclusion criteria were peer-reviewed published randomized clinical trials that compared simulators versus standard apprenticeship surgical training of surgical trainees with little or no prior laparoscopic experience. Of the 551 relevant studies found, 17 trials fulfilled all inclusion criteria. The effect sizes (ES) with 95 % confidence intervals [CI] were calculated for multiple psychometric skill outcome measures. RESULTS: Data were combined by means of both fixed- and random-effects models. Meta-analytic combined effect size estimates showed that novice students who trained on simulators were superior in their performance and skill scores (d = 1.98, 95 % CI: 1.20-2.77; P < 0.01), were more careful in handling various body tissue (d = 1.08, 95 % CI: 0.36-1.80; P < 0.01), and had a higher accuracy score in conducting laparoscopic tasks (d = 1.38, 95 % CI: 0.30-2.47; P < 0.05). CONCLUSION: Simulators have been shown to provide better laparoscopic surgery skills training for trainees than the traditional standard apprenticeship approach to skill development. Surgical residency programs are highly encouraged to adopt the use of simulators in teaching laparoscopic surgery skills to novice students.

A review of student evaluation of teaching: applications to veterinary medical education.

Student evaluation of teaching is ubiquitous to teaching in colleges and universities around the world. Since the implementation of student evaluations in the 1970s in the US, considerable research has been devoted to their appropriate use as a means of judging the effectiveness of teaching. The present article aims to (1) examine the evidence for the reliability, validity, and utility of student ratings; (2) provide seven guidelines for ways to identify effective instruction, given that the purpose of student evaluation is to assess effective teaching; and (3) conclude with recommendations for the integration of student ratings into the continuous evaluation of veterinary medical education.

Student and teaching characteristics related to ratings of instruction in medical sciences graduate programs.

BACKGROUND: Although the validity of students' ratings of instruction has been documented, several student and course characteristics may be related to the ratings students give their instructors. AIMS: The purpose of this study was to examine student ratings obtained from the Universal Student Ratings of Instruction (USRI) instrument. These responses were compared to various student characteristics. Also, teaching characteristics that were most closely associated with the ratings were determined. METHOD: A total of 1738 USRI forms were completed by graduate students enrolled in medical science courses from 1999 to 2006 in the Faculty of Medicine at a Canadian university. RESULTS: Between group comparisons showed that negative student perceptions about the course (i.e., did not have the freedom to select), perceiving the course workload as high, and low grade expectations held were related to negative student ratings of overall quality of instruction. In terms of the student and teaching characteristics, organization of course material and perceptions of whether students felt they learned a lot in the course were most closely related to global ratings of instructional quality. CONCLUSION: Implications for teaching focus on improving the organization and delivery of course content that meets the learning objectives of graduate students in medical sciences.

A teaching model for aromatase inhibitors after tamoxifen.

Breast cancer is the most common cancer diagnosed in women. The present study evaluated the family physicians' (FPs) understanding of adjuvant hormonal therapies for an early breast cancer. FPs were invited to attend teaching workshops on this topic, which utilized a pretest, didactic and interactive teaching, and posttest format. FPs (n = 23) showed an improvement (p < 0.001) in pretest to posttest score. It is clear that, with a targeted teaching, FPs can quickly become more knowledgeable on the topic of hormonal therapies in breast cancer, with the potential of applying this information in their own practice.

Assessing ethical problem solving by reasoning rather than decision making.

CONTEXT: The assessment of ethical problem solving in medicine has been controversial and challenging. The purposes of this study were: (i) to create a new instrument to measure doctors' decisions on and reasoning approach towards resolving ethical problems; (ii) to evaluate the scores generated by the new instrument for their reliability and validity, and (iii) to compare doctors' ethical reasoning abilities between countries and among medical students, residents and experts. METHODS: This study used 15 clinical vignettes and the think-aloud method to identify the processes and components involved in ethical problem solving. Subjects included volunteer ethics experts, postgraduate Year 2 residents and pre-clerkship medical students. The interview data were coded using the instruments of the decision score and Ethical Reasoning Inventory (ERI). The ERI assessed the quality of ethical reasoning for a particular case (Part I) and for an individual globally across all the vignettes (Part II). RESULTS: There were 17 Canadian and 32 Taiwanese subjects. Based on the Canadian standard, the decision scores between Taiwanese and Canadian subjects differed significantly, but made no discrimination among the three levels of expertise. Scores on the ERI Parts I and II, which reflect doctors' reasoning quality, differed between countries and among different levels of expertise in Taiwan, providing evidence of construct validity. In addition, experts had a greater organised knowledge structure and considered more relevant variables in the process of arriving at ethical decisions than did residents or students. The reliability of ERI scores was 0.70-0.99 on Part I and 0.75-0.80 on Part II. CONCLUSIONS: Expertise in solving ethical problems could not be differentiated by the decisions made, but could be differentiated according to the reasoning used to make those decisions. The difference between Taiwanese and Canadian experts suggests that cultural considerations come into play in the decisions that are made in the course of providing humane care to patients.

A predictive validity study of medical judgment vignettes to assess students' noncognitive attributes: a 3-year prospective longitudinal study.

BACKGROUND: The admissions interview still remains the most common approach used to describe candidates' noncognitive attributes for medical school. AIM: In this prospective study, we have investigated the predictive validity of a semi-structured interview for admissions to medical school based on medical judgment vignettes: (1) ethical decision-making (moral), (2) relationships with patients and their families (altruistic), and (3) roles and responsibilities in professional relationships (dutiful). METHOD: A group of 26 medical students from the Class of 2007 participated in the interview process and provided their subsequent performance results from clerkship 3 years later. RESULTS: Inter-rater reliability of the scored interviews was high (kappa = 0.96). Our results provided evidence for both convergent and divergent predictive validity. Medical judgment vignettes scores correlated significantly with seven mandatory clerkship rotation in-training evaluation reports (r = 0.39, p < 0.05; to r = 0.55, p < 0.01). CONCLUSION: This semi-structured interview based on clearly defined and scored medical judgment vignettes that focus on the assessment of medical students' noncognitive attributes is promising for student's selection into medical school. The high reliability and evidence of predictive validity of clinical performance over a 3-year period suggests a workable approach to the assessment of 'compelling personal characteristics' beyond merely cognitive variables.

Assessment of applicants to the veterinary curriculum using a multiple mini-interview method.

This study describes the development, implementation, and psychometric assessment of the multiple mini-interview (MMI) for the inaugural class of veterinary medicine applicants at the University of Calgary Faculty of Veterinary Medicine (UCVM). The MMI is a series of approximately five to 12 10-minute interviews that consist of situational events. Applicants are given a scenario and asked to work through an issue or behavioral-type questions that are meant to assess one attribute (e.g., empathy) at a time. This structure allows for multiple assessments of the applicants by trained interviewers on the same questions. MMI scenario development was based on a review of the noncognitive attributes currently assessed by the 31 veterinary schools across Canada and the United States and the goals and objectives of UCVM. The noncognitive attributes of applicants (N=110) were assessed at five stations, by two interviewers within each station, on three items using a standardized rating form on an anchored 1-5 scale. The method was determined to be reliable (G-coefficient=0.88) and demonstrated evidence of validity. The MMI score did not correlate with grade-point average (r=0.12, p=0.22). While neither the applicants nor interviewers had participated in an MMI format before, both groups reported the process to be acceptable in a post-interview questionnaire. This analysis provides preliminary evidence of the reliability, validity, and acceptability of the MMI in assessing the noncognitive attributes of applicants for veterinary medical school admissions.

So much to teach, so little time: a prospective cohort study evaluating a tool to select content for a critical care curriculum.

INTRODUCTION: Curricular content is often based on the personal opinions of a small number of individuals. Although convenient, such curricula may not meet the needs of the target learner, the program or the institution. Using an objective method to ensure content validity of a curriculum can alleviate this issue. METHODS: A form was created that listed clinical presentations relevant to residents completing intensive care unit (ICU) rotations. Twenty residents and 20 intensivists in tertiary academic multisystem ICUs ranked each presentation on three separate scales: how life-threatening each is, how commonly each is seen in critical care, and how reversible each is. Mean scores for the individual scales were calculated, and these three values were subsequently multiplied together to achieve a composite score for each presentation. The correlation between the two groups' scores for the presentations was calculated to assess reliability of the process. RESULTS: There was excellent agreement between the two groups for rating each presentation (correlation coefficient r = 0.94). The 10 clinical presentations with the highest composite scores formed the basis of our new curriculum. CONCLUSIONS: We describe a method that can be used to select the content of a curriculum for learners in an ICU. Although the content that we selected to include in our curriculum may not be applicable to other ICUs, we believe that the process we used is easily applied elsewhere, and that it provides an efficient method to improve content validity of a curriculum.

Procedural skills education--colonoscopy as a model.

Traditionally, surgical and procedural apprenticeship has been an assumed activity of students, without a formal educational context. With increasing barriers to patient and operating room access such as shorter work week hours for residents, and operating room and endoscopy time at a premium, alternate strategies to maximizing procedural skill development are being considered. Recently, the traditional surgical apprenticeship model has been challenged, with greater emphasis on the need for surgical and procedural skills training to be more transparent and for alternatives to patient-based training to be considered. Colonoscopy performance is a complex psychomotor skill requiring practitioners to integrate multiple sensory inputs, and involves higher cortical centres for optimal performance. Colonoscopy skills involve mastery in the cognitive, technical and process domains. In the present review, we propose a model for teaching colonoscopy to the novice trainee based on educational theory.

A generalizability study of the medical judgment vignettes interview to assess students' noncognitive attributes for medical school.

BACKGROUND: Although the reliability of admission interviews has been improved through the use of objective and structured approaches, there still remains the issue of identifying and measuring relevant attributes or noncognitive domains of interest. In this present study, we use generalizability theory to determine the estimated variance associated with participants, judges and stations from a semi-structured, Medical Judgment Vignettes interview used as part of an initiative to improve the reliability and content validity of the interview process used in the selection of students for medical school. METHODS: A three station, Medical Judgment Vignettes interview was conducted with 29 participants and scored independently by two judges on a well-defined 5-point rubric. Generalizability Theory provides a method for estimating the variability of a number of facets. In the present study each judge (j) rated each participant (p) on all three Medical Judgment Vignette stations (s). A two-facet crossed designed generalizability study was used to determine the optimal number of stations and judges to achieve a 0.80 reliability coefficient. RESULTS: The results of the generalizability analysis showed that a three station, two judge Medical Judgment Vignettes interview results in a G coefficient of 0.70. As shown by the adjusted E rho 2 scores, since interviewer variability is negligible, increasing the number of judges from two to three does not improve the generalizability coefficient. Increasing the number of stations, however, does have a substantial influence on the overall dependability of this measurement. In a decision study analysis, increasing the number of stations to six with a single judge at each station results in a G coefficient of 0.81. CONCLUSION: The Medical Judgment Vignettes interview provides a reliable approach to the assessment of candidates' noncognitive attributes for medical school. The high inter-rater reliability is attributed to the greater objectivity achieved through the used of the semi-structured interview format and clearly defined scoring rubric created for each of the judgment vignettes. Despite the relatively high generalizability coefficient obtained for only three stations, future research should further explore the reliability, and equally importantly, the validity of the vignettes with a large group of candidates applying for medical school.

The need for national medical licensing examination in Saudi Arabia.

BACKGROUND: Medical education in Saudi Arabia is facing multiple challenges, including the rapid increase in the number of medical schools over a short period of time, the influx of foreign medical graduates to work in Saudi Arabia, the award of scholarships to hundreds of students to study medicine in various countries, and the absence of published national guidelines for minimal acceptable competencies of a medical graduate. DISCUSSION: We are arguing for the need for a Saudi national medical licensing examination that consists of two parts: Part I (Written) which tests the basic science and clinical knowledge and Part II (Objective Structured Clinical Examination) which tests the clinical skills and attitudes. We propose this examination to be mandated as a licensure requirement for practicing medicine in Saudi Arabia. CONCLUSION: The driving and hindering forces as well as the strengths and weaknesses of implementing the licensing examination are discussed in details in this debate.

The predictive validity of the MCAT for medical school performance and medical board licensing examinations: a meta-analysis of the published research.

PURPOSE: To conduct a meta-analysis of published studies to determine the predictive validity of the MCAT on medical school performance and medical board licensing examinations. METHOD: The authors included all peer-reviewed published studies reporting empirical data on the relationship between MCAT scores and medical school performance or medical board licensing exam measures. Moderator variables, participant characteristics, and medical school performance/medical board licensing exam measures were extracted and reviewed separately by three reviewers using a standardized protocol. RESULTS: Medical school performance measures from 11 studies and medical board licensing examinations from 18 studies, for a total of 23 studies, were selected. A random-effects model meta-analysis of weighted effects sizes (r) resulted in (1) a predictive validity coefficient for the MCAT in the preclinical years of r = 0.39 (95% confidence interval [CI], 0.21-0.54) and on the USMLE Step 1 of r = 0.60 (95% CI, 0.50-0.67); and (2) the biological sciences subtest as the best predictor of medical school performance in the preclinical years (r = 0.32 95% CI, 0.21-0.42) and on the USMLE Step 1 (r = 0.48 95% CI, 0.41-0.54). CONCLUSIONS: The predictive validity of the MCAT ranges from small to medium for both medical school performance and medical board licensing exam measures. The medical profession is challenged to develop screening and selection criteria with improved validity that can supplement the MCAT as an important criterion for admission to medical schools.