Considerations that will determine if competency-based assessment is a sustainable innovation.

Educational assessment for the health professions has seen a major attempt to introduce competency based frameworks. As high level policy developments, the changes were intended to improve outcomes by supporting learning and skills development. However, we argue that previous experiences with major innovations in assessment offer an important road map for developing and refining assessment innovations, including careful piloting and analyses of their measurement qualities and impacts. Based on the literature, numerous assessment workshops, personal interactions with potential users, and our 40 years of experience in implementing assessment change, we lament the lack of a coordinated approach to clarify and improve measurement qualities and functionality of competency based assessment (CBA). To address this worrisome situation, we offer two roadmaps to guide CBA's further development. Initially, reframe and address CBA as a measurement development opportunity. Secondly, using a roadmap adapted from the management literature on sustainable innovation, the medical assessment community needs to initiate an integrated plan to implement CBA as a sustainable innovation within existing educational programs and self-regulatory enterprises. Further examples of down-stream opportunities to refocus CBA at the implementation level within faculties and within the regulatory framework of the profession are offered. In closing, we challenge the broader assessment community in medicine to step forward and own the challenge and opportunities to reframe CBA as an innovation to improve the quality of the clinical educational experience. The goal is to optimize assessment in health education and ultimately improve the public's health.

Patients of doctors further from medical school graduation have poorer outcomes.

CONTEXT: There is an apparent contradiction between the findings of studies indicating that patient outcomes are better when physicians have a greater volume of practice and those that find outcomes to be worse with increased time since training, which implies greater volume. OBJECTIVES: This study was designed to estimate the adjusted relationships between physicians' characteristics, including recent practice volume and time since medical school graduation, and patient outcomes. METHODS: This is a retrospective observational study based on all Pennsylvania hospitalisations over 7 years for acute myocardial infarction, congestive heart failure, gastrointestinal haemorrhage, hip fracture and pneumonia. It refers to 694 020 hospitalisations in 184 hospitals attended by 5280 internists and family physicians. Patient severity of illness at admission and in-hospital mortality, hospital location and volume, and the physician's recent practice volume, time since medical school graduation, board certification, and citizenship or medical school location were analysed. RESULTS: After adjustment, recent practice volume did not have a statistically significant association with in-hospital mortality for all of the conditions combined. By contrast, each decade since graduation from medical school was associated with a 4.5% increase in relative risk for patient mortality. CONCLUSIONS: Recent practice volume does not mitigate the increase in patient mortality associated with physicians' time since medical school graduation. These findings underscore the need to finds ways to support and encourage learning.

Introducing a model for optimal design of sequential objective structured clinical examinations.

In a sequential OSCE which has been suggested to reduce testing costs, candidates take a short screening test and who fail the test, are asked to take the full OSCE. In order to introduce an effective and accurate sequential design, we developed a model for designing and evaluating screening OSCEs. Based on two datasets from a 10-station pre-internship OSCE and considering three factors, namely, the number of stations, the criteria for selecting the stations, and the cut-off score, several hypothetical tests were proposed. To investigate their accuracy, the positive predictive value (PPV), the pass rate, and the negative predictive value (NPV) were calculated. Also, a "desirable" composite outcome was defined as PPV = 100 %, pass rate ≥50 %, and NPV ≥25 %. Univariate and multiple logistic regression analyses were conducted to estimate the effects of independent factors on the occurrence of the desirable outcome. In half of the screening tests no false positive result was detected. Most of the screening OSCEs had acceptable levels of pass rate and NPV. Considering the desirable composite outcome 20 screening OSCEs could have successfully predicted the results of the corresponding full OSCE. The multiple regression analysis indicated significant contributions for the selection criteria (p values = 0.019) and the cut-off score (p values = 0.017). In order to have efficient screening OSCEs with the lowest probability of the error rate, careful selection of stations with high values of discrimination or item total correlation, and use of a relatively stringent cut-off score should be considered.

Competency Evaluations in the Next Accreditation System: Contributing to Guidelines and Implications.

UNLABELLED: CONSTRUCT: This study examines validity evidence of end-of-rotation evaluation scores used to measure competencies and milestones as part of the Next Accreditation System (NAS) of the Accreditation Council for Graduate Medical Education (ACGME). BACKGROUND: Since the implementation of the milestones, end-of-rotation evaluations have surfaced as a potentially useful assessment method. However, validity evidence on the use of rotation evaluation scores as part of the NAS has not been studied. This article examines validity evidence for end-of-rotation evaluations that can contribute to developing guidelines that support the NAS. APPROACH: Data from 2,701 end-of-rotation evaluations measuring 21 out of 22 Internal Medicine milestones for 142 residents were analyzed (July 2013-June 2014). Descriptive statistics were used to measure the distribution of ratings by evaluators (faculty, n = 116; fellows, n = 59; peer-residents, n = 131), by postgraduate years. Generalizability analysis and higher order confirmatory factor analysis were used to examine the internal structure of ratings. Psychometric implications for combining evaluation scores using composite score reliability were examined. RESULTS: Milestone ratings were significantly higher for each subsequent year of training (15/21 milestones). Faculty evaluators had greater variability in ratings across milestones, compared to fellows and residents; faculty ratings were generally correlated with milestone ratings from fellows (r = .45) and residents (r = .25), but lower correlations were found for Professionalism and Interpersonal and Communication Skills. The Φ-coefficient was .71, indicating good reliability. Internal structure supported a 6-factor solution, corresponding to the hierarchical relationship between the milestones and the 6 core competencies. Evaluation scores corresponding to Patient Care, Medical Knowledge, and Practice-Based Learning and Improvement had higher correlations to milestones reported to the ACGME. Mean evaluation ratings predicted problem residents (odds ratio = 5.82, p < .001). CONCLUSIONS: Guidelines for rotation evaluations proposed in this study provide useful solutions that can help program directors make decisions on resident progress and contribute to assessment systems in graduate medical education.

Implementation of competency-based medical education: are we addressing the concerns and challenges?

CONTEXT: Competency-based medical education (CBME) has emerged as a core strategy to educate and assess the next generation of physicians. Advantages of CBME include: a focus on outcomes and learner achievement; requirements for multifaceted assessment that embraces formative and summative approaches; support of a flexible, time-independent trajectory through the curriculum; and increased accountability to stakeholders with a shared set of expectations and a common language for education, assessment and regulation. OBJECTIVES: Despite the advantages of CBME, numerous concerns and challenges to the implementation of CBME frameworks have been described, including: increased administrative requirements; the need for faculty development; the lack of models for flexible curricula, and inconsistencies in terms and definitions. Additionally, there are concerns about reductionist approaches to assessment in CBME, lack of good assessments for some competencies, and whether CBME frameworks include domains of current importance. This study will outline these issues and discuss the responses of the medical education community. METHODS: The concerns and challenges expressed are primarily categorised as: (i) those related to practical, administrative and logistical challenges in implementing CBME frameworks, and (ii) those with more conceptual or theoretical bases. The responses of the education community to these issues are then summarised. CONCLUSIONS: The education community has begun to address the challenges involved in implementing CBME. Models and guidance exist to inform implementation strategies across the continuum of education, and focus on the more efficient use of resources and technology, and the use of milestones and entrustable professional activities-based frameworks. Inconsistencies in CBME definitions and frameworks remain a significant obstacle. Evolution in assessment approaches from in vitro task-based methods to in vivo integrated approaches is responsive to many of the theoretical and conceptual concerns about CBME, but much work remains to be done to bring rigour and quality to work-based assessment.

How to set standards on performance-based examinations: AMEE Guide No. 85.

This AMEE Guide offers an overview of methods used in determining passing scores for performance-based assessments. A consideration of various assessment purposes will provide context for discussion of standard setting methods, followed by a description of different types of standards that are typically set in health professions education. A step-by-step guide to the standard setting process will be presented. The Guide includes detailed explanations and examples of standard setting methods, and each section presents examples of research done using the method with performance-based assessments in health professions education. It is intended for use by those who are responsible for determining passing scores on tests and need a resource explaining methods for setting passing scores. The Guide contains a discussion of reasons for assessment, defines standards, and presents standard setting methods that have been researched with performance-based tests. The first section of the Guide addresses types of standards that are set. The next section provides guidance on preparing for a standard setting study. The following sections include conducting the meeting, selecting a method, implementing the passing score, and maintaining the standard. The Guide will support efforts to determine passing scores that are based on research, matched to the assessment purpose, and reproducible.

Outcomes of cardiac surgery: associations with physician characteristics, institutional characteristics, and transfers of care.

BACKGROUND: Although there are several studies of the human and system factors that influence the outcomes of cardiac surgery, it remains difficult to draw conclusions because many do not simultaneously adjust for the characteristics of patients, physicians, and institutions. The current study explores the associations between these factors and inhospital mortality, with a particular focus on whether patients had the same operating and attending physician. METHOD AND RESULTS: This is a retrospective observational study of 114,751 hospitalizations from 2003 to 2009 in Pennsylvania that included a coronary artery bypass graft, valve surgery, or both. The study included 70 teaching and nonteaching hospitals, 289 operating physicians who were also the attending physicians for 75% of the hospitalizations, and 2654 attending physicians for the remaining hospitalizations. After adjustment, there was a 38.4% decrease (95% CI, 20.3%-56.5%) in mortality when the operating and attending physician were the same. For the operator, each procedure performed was associated with a 0.05% (95% CI, 0.04%-0.06%) decrease in mortality and each year since medical school was associated with a 0.9% (95% CI, 0.02%-1.8%) increase in mortality. For the attending, each year since medical school was associated with a 0.67% (95% CI, 0.01%-1.4%) decrease in patient mortality. CONCLUSIONS: The findings indicated that an increase in the log odds of mortality was associated with the transfer of care between an attending and operating physician. Better patient outcomes were associated with an operator with higher volume who was closer to medical school graduation and an attending physician with more clinical experience.

Assessment in the context of licensure and certification.

Over the past 25 years, three major forces have had a significant influence on licensure and certification: the shift in focus from educational process to educational outcomes, the increasing recognition of the need for learning and assessment throughout a physician's career, and the changes in technology and psychometrics that have opened new vistas for assessment. These forces have led to significant changes in assessment for licensure and certification. To respond to these forces, licensure and certification programs have improved the ways in which their examinations are constructed, scored, and delivered. In particular, we note the introduction of adaptive testing; automated item creation, scoring, and test assembly; assessment engineering; and data forensics. Licensure and certification programs have also expanded their repertoire of assessments with the rapid development and adoption of simulation and workplace-based assessment. Finally, they have invested in research intended to validate their programs in four ways: (a) the acceptability of the program to stakeholders, (b) the extent to which stakeholders are encouraged to learn and improve, (c) the extent to which there is a relationship between performance in the programs and external measures, and (d) the extent to which there is a relationship between performance as measured by the assessment and performance in practice. Over the past 25 years, changes in licensure and certification have been driven by the educational outcomes movement, the need for lifelong learning, and advances in technology and psychometrics. Over the next 25 years, we expect these forces to continue to exert pressure for change which will lead to additional improvement and expansion in examination processes, methods of assessment, and validation research.

Newly qualified doctors' views on the significance and accessibility of career advice during medical training in Saudi Arabia.

BACKGROUND: Career advice is an important instrument to help students with the proper specialty selection. The study aims (1) to explore the views of newly graduated doctors in Saudi Arabia about their experience with the current status of career support system during medical training and (2) to identify cross-cultural similarities and differences. METHODS: A cross-sectional design study was conducted using a questionnaire to elicit the responses of participants from newly qualified doctors concerning the availability and significance of career advice. SPSS (version 11.0; Chicago, IL) was used to analyze the data and statistical tests, such as chi-square and unpaired t tests, were used to analyze the observations. RESULTS: A response rate of 94.7% was obtained. Among this group, 102 were males and 78 were females. Only 53% did receive career advice. The majority of men felt that career advice during medical studies was inadequate, while women were less negative (69% versus 32%; p = 0.0001). Furthermore, men were more disappointed about the possibilities for career advice after graduating than women (34% versus 13%, p = 0.0001). CONCLUSIONS: The results show that only half of newly graduated doctors had received any career advice during medical training. As the health care system cannot afford the potential waste of time and resources for doctors, career guidance should begin in undergraduate training so that the process of thinking about their future career starts longtime before they make their career choice.

Associations between initial American Board of Internal Medicine certification and maintenance of certification status of attending physicians and in-hospital mortality of patients with acute myocardial infarction or congestive heart failure: a retrospective cohort study of hospitalisations in Pennsylvania, USA.

OBJECTIVE: To determine whether internists' initial specialty certification and the maintenance of that certification (MOC) is associated with lower in-hospital mortality for their patients with acute myocardial infarction (AMI) or congestive heart failure (CHF). DESIGN: Retrospective cohort study of hospitalisations in Pennsylvania, USA, from 2012 to 2017. SETTING: All hospitals in Pennsylvania. PARTICIPANTS: All 184 115 hospitalisations for primary diagnoses of AMI or CHF where the attending physician was a self-designated internist. PRIMARY OUTCOME MEASURE: In-hospital mortality. RESULTS: Of the 2575 physicians, 2238 had initial certification and 820 were eligible for MOC. After controlling for patient demographics and clinical characteristics, hospital-level factors and physicians' demographic and medical school characteristics, both initial certification and MOC were associated with lower mortality. The adjusted OR for initial certification was 0.835 (95% CI 0.756 to 0.922; p<0.001). Patients cared for by physicians with initial certification had a 15.87% decrease in mortality compared with those cared for by non-certified physicians (mortality rate difference of 5.09 per 1000 patients; 95% CI 2.12 to 8.05; p<0.001). The adjusted OR for MOC was 0.804 (95% CI 0.697 to 0.926; p=0.003). Patients cared for by physicians who completed MOC had an 18.91% decrease in mortality compared with those cared for by MOC lapsed physicians (mortality rate difference of 6.22 per 1000 patients; 95% CI 2.0 to 10.4; p=0.004). CONCLUSIONS: Initial certification was associated with lower mortality for AMI or CHF. Moreover, for patients whose physicians had initial certification, an additional advantage was associated with its maintenance.

Comparison of two methods of standard setting: the performance of the three-level Angoff method.

CONTEXT: Cut-scores, reliability and validity vary among standard-setting methods. The modified Angoff method (MA) is a well-known standard-setting procedure, but the three-level Angoff approach (TLA), a recent modification, has not been extensively evaluated. OBJECTIVES: This study aimed to compare standards and pass rates in an objective structured clinical examination (OSCE) obtained using two methods of standard setting with discussion and reality checking, and to assess the reliability and validity of each method. METHODS: A sample of 105 medical students participated in a 14-station OSCE. Fourteen and 10 faculty members took part in the MA and TLA procedures, respectively. In the MA, judges estimated the probability that a borderline student would pass each station. In the TLA, judges estimated whether a borderline examinee would perform the task correctly or not. Having given individual ratings, judges discussed their decisions. One week after the examination, the procedure was repeated using normative data. RESULTS: The mean score for the total test was 54.11% (standard deviation: 8.80%). The MA cut-scores for the total test were 49.66% and 51.52% after discussion and reality checking, respectively (the consequent percentages of passing students were 65.7% and 58.1%, respectively). The TLA yielded mean pass scores of 53.92% and 63.09% after discussion and reality checking, respectively (rates of passing candidates were 44.8% and 12.4%, respectively). Compared with the TLA, the MA showed higher agreement between judges (0.94 versus 0.81) and a narrower 95% confidence interval in standards (3.22 versus 11.29). CONCLUSIONS: The MA seems a more credible and reliable procedure with which to set standards for an OSCE than does the TLA, especially when a reality check is applied.

Increasing the quality and capacity of education: the challenge for the 21st century.

CONTEXT: Abraham Flexner's 1910 report to the Carnegie Foundation was a successful attempt to improve the quality of health care by reforming the education of health care providers. It was accompanied by a significant reduction in the number of medical schools and an increase in the quality of those schools that remained. Although the report's focus on quality was laudable and appropriate to the times, we now face a significant shortage and maldistribution of health care workers, particularly in countries with the highest burden of disease. Hence, we see the challenges for the 21st century to involve increasing both capacity and quality. DISCUSSION: In our view, these two goals can be achieved through three research-driven educational reforms. Firstly, many educational methodologies are retained based on tradition and new methods are adopted based on fashion. Educational research must become the basis for educational practice. Secondly, educational methodology is often focused on improving quality and does not consider resource utilisation, which reduces its relevance and utility. Educational research must focus on quality and efficiency. Thirdly, one form of educational quality control is provided by accreditation processes. Some of these processes are so prescriptive that they are a barrier to improvement and for none is there evidence of effectiveness. Accreditation processes should be based on data about what is effective and efficient. CONCLUSIONS: Just as Flexner argued for a scientific basis in the practice of medicine, we argue for a scientific basis in the practice of education. In our view, this is the way to meet the challenges of the 21st century.

We Must Graduate Physicians, Not Doctors.

Today, medical schools graduate doctors, not physicians. Thousands of doctors who are U.S. citizens and graduates of U.S. and international medical schools will never become physicians because they do not obtain a residency position. Doctors need at least one year of residency to become a licensed physician. However, 4,099 applicants in 2018 and 4,170 in 2019 failed to get a position through the National Resident Matching Program Main Match; about 1,000 students get positions after the Main Match each year. The personal and societal cost is enormous: each year, approximately 3,000 nonphysician doctors cannot use 12,000 education years and three-quarters of a billion dollars they invested in medical education and cannot mitigate the shortfall of 112,000 physicians expected in 2030.To ameliorate this problem, medical schools could guarantee one year of residency. This is affordable: despite federally funded slots being capped, residency positions have increased for 17 consecutive years (20,602 in 2002 to 32,194 in 2019) because residents are cost-effective additions to the workforce. Alternatively, a 3-year curriculum plus required fourth-year primary care residency is another option. The salary during the residency year could equal other first-year residents', or there could be a token amount for this "internship." Both models decrease the cost of medical education; the second financially unburdens the hospital.Since the Flexner Report (when there was no formal postgraduate training), the end point of medical education has moved from readiness for independent medical practice (physician) to readiness for postgraduate training (doctor). To benefit individuals and society, medical education must take steps to ensure that all graduates are physicians, not just doctors.

Specialty Board Certification Rate as an Outcome Metric for GME Training Institutions: A Relationship With Quality of Care.

Educational outcome measures, known to be associated with the quality of care, are needed to support improvements in graduate medical education (GME). This retrospective observational study sought to determine whether there was a relationship between the specialty board certification rates of GME training institutions and the quality of care delivered by their graduates. It is based on 7 years of hospitalizations in Pennsylvania (N = 354,767) with diagnoses of acute myocardial infarction, congestive heart failure, gastrointestinal hemorrhage, or pneumonia. The 2,265 attending physicians were self-identified internists, and they completed their training in 59 institutions. The percentage of board-certified physicians from each training institution, excluding the physician herself or himself, was calculated and an indicator of whether it exceeded 80% was created. This was analyzed against inhospital mortality and length of stay, adjusted for patient/physician/hospital characteristics. There were significantly lower odds of mortality (adjusted Odd's ratio [OR] = .92, 95% CI [0.86, 0.98]) and log length of stay (adjusted OR = .98, 95% CI [.94, .99]) when the attending physician trained in a residency program with an 80% or greater certification rate. The results suggest that specialty certification rates may be a useful educational outcome for residency training programs.

Setting standards for teaching evaluation data: an application of the contrasting groups method.

BACKGROUND: Teaching evaluations are widely used in retention and remediation decisions. Typically evaluations are reviewed in a global manner and some gestalt is reached. PURPOSE: Apply the Contrasting Groups standard setting methodology to faculty teaching dossiers, to examine resulting "pass-rates" and precision of the decisions. METHODS: Ten faculty judges set standards for teaching dossiers using a Contrasting Groups approach. Blinded dossiers summarizing clinical (N = 47) and classroom teaching (N = 37) were sorted into piles labeled Unsatisfactory, Satisfactory, Excellent, and Superior. Cut-points were the midpoints between aggregated judge-level mean performances of dossiers placed within adjacent levels. RESULTS: For the total faculty, the percentage assigned to groups labeled Unsatisfactory through Superior, were 4.1%, 5.9%, 26.1%, and 63.9% and 6.6%, 24.8%, 44.1%, and 24.5% for clinical and classroom teaching respectively. Standard error of measurement was between .20 and .25 with 5 to 7 judges. CONCLUSIONS: Standard setting methods applied to faculty evaluation data produce precise results. Future work can examine standard stability and acceptability, and methods to combine across teaching venues.

Pathways to Independent Primary Care Clinical Practice: How Tall Is the Shortest Giant?

Patients can be treated by a physician, a nurse practitioner (NP), or a physician assistant (PA) despite marked differences in the education and training for these three professions. This natural experiment allows examination of a critical question: What is the minimum education and training required to practice primary care? In other words, how tall is the shortest giant? State licensing requirements, not educational bodies, legislate minimum training. The current minimum is 6 years, which includes 27.5 weeks of supervised clinical experience (SCE), for NPs. In comparison, PAs train for 6 years with 45 weeks of SCE, and physicians for at least 8 years with 110 weeks of SCE. Initial, flawed studies show equivalent patient outcomes among the professions. If rigorous follow-up studies confirm equivalence, the content and length of medical education for primary care physicians should be reconsidered. Unmatched medical school graduates, with 7 years of training and 65 weeks of SCE, more than the required minimum for NPs, deserve to practice independently. So do PAs. If equivalence is not confirmed, the minimum requirements for NPs and/or PAs should be raised, including considering a required residency (currently optional). Alternatively, the scope of practice for the three professions could be defined to reflect differences in training. There is an urgent need to set aside preconceived notions and turf battles, conduct rigorous independent studies, and generate meaningful data on practice patterns and patient outcomes. This should inform optimal training, scope of practice, and workforce development for each invaluable primary care clinical practitioner.

Interprofessional communication in a socio-hierarchical culture: development of the TRI-O guide.

OBJECTIVES: Interprofessional education (IPE) and collaborative practice are essential for patient safety. Effective teamwork starting with partnership-based communications should be introduced early in the educational process. Many societies in the world hold socio-hierarchical culture with a wide power distance, which makes collaboration among health professionals challenging. Since an appropriate communication framework for this context is not yet available, this study filled that gap by developing a guide for interprofessional communication, which is best suited to the socio-hierarchical and socio-cultural contexts. MATERIALS AND METHODS: The draft of the guide was constructed based on previous studies of communication in health care in a socio-hierarchical context, referred to international IPE literature, and refined by focus group discussions among various health professionals. Nominal group technique, also comments from national and international experts of communication skills in health care, was used to validate the guide. A pilot study with a pre-posttest design was conducted with 53 first- and 107 fourth-year undergraduate medical, nursing, and health nutrition students. RESULTS: We developed the "TRI-O" guide of interprofessional communication skills, emphasizing "open for collaboration, open for information, open for discussion", and found that the application of the guide during training was feasible and positively influenced students' perceptions. CONCLUSION: The findings suggest that the TRI-O guide is beneficial to help students initiate partnership-based communication and mutual collaboration among health professionals in the socio-hierarchical and socio-cultural context.

Assessing clinical reasoning skills using Script Concordance Test (SCT) and extended matching questions (EMQs): A pilot for urology trainees.

INTRODUCTION: Clinical reasoning skill is the core of medical competence. Commonly used assessment methods for medical competence have limited ability to evaluate critical thinking and reasoning skills. Script Concordance Test (SCT) and Extended Matching Questions (EMQs) are the evolving tests which are considered to be valid and reliable tools for assessing clinical reasoning and judgment. We performed this pilot study to determine whether SCT and EMQs can differentiate clinical reasoning ability among urology residents, interns and medical students. METHODS: This was a cross-sectional study in which an examination with 48 SCT-based items on eleven clinical scenarios and four themed EMQs with 21 items were administered to a total of 27 learners at three differing levels of experience i.e. 9 urology residents, 6 interns and 12 fifth year medical students. A non-probability convenience sampling was done. The SCTs and EMQs were developed from clinical situations representative of urological practice by 5 content experts (urologists) and assessed by a medical education expert. Learners' responses were scored using the standard and the graduated key. A one way analysis of variance (ANOVA) was conducted to compare the mean scores across the level of experience. A p-value of < 0.05 was considered statistically significant. Test reliability was estimated by Cronbach α. A focused group discussion with candidates was done to assess their perception of test. RESULTS: Both SCT and EMQs successfully differentiated residents from interns and students. Statistically significant difference in mean score was found for both SCT and EMQs among the 3 groups using both the standard and the graduated key. The mean scores were higher for all groups as measured by the graduated key compared to the standard key. The internal consistency (Cronbach's α) was 0.53 and 0.6 for EMQs and SCT, respectively. Majority of the participants were satisfied with regard to time, environment, instructions provided and the content covered and nearly all felt that the test helped them in thinking process particularly clinical reasoning. CONCLUSIONS: Our data suggest that both SCT and EMQs are capable of discriminating between learners according to their clinical experience in urology. As there is a wide acceptability by all candidates, these tests could be used to assess and enhance clinical reasoning skills. More research is needed to prove validity of these tests.