A Proposed Blueprint for Operative Performance Training, Assessment, and Certification.

OBJECTIVE: The aim of this study was to propose an evidence-based blueprint for training, assessment, and certification of operative performance for surgical trainees. SUMMARY BACKGROUND DATA: Operative skill is a critical aspect of surgical performance. High-quality assessment of operative skill therefore has profound implications for training, accreditation, certification, and the public trust of the profession. Current methods of operative skill assessment for surgeons rely heavily on global assessment strategies across a very broad domain of procedures. There is no mechanism to assure technical competence for individual procedures. The science and scalability of operative skill assessment has progressed significantly in recent decades, and can inform a much more meaningful strategy for competency-based assessment of operative skill than has been previously achieved. METHODS: The present article reviews the current status and science of operative skill assessment and proposes a template for competency-based assessment which could be used to update training, accreditation, and certification processes. The proposal is made in reference to general surgery but is more generally applicable to other procedural specialties. RESULTS: Streamlined, routine assessment of every procedure performed by surgical trainees is feasible and would enable a more competency-based educational paradigm. In light of the constraints imposed by both clinical volume and assessment bias, trainees should be expected to become proficient and be measured against a mastery learning standard only for the most important and highest-frequency procedures. For less frequently observed procedures, performance can be compared to a norm-referenced standard and, to provide an overall trajectory of performance, analyzed in aggregate. Key factors in implementing this approach are the number of evaluations, the number of raters, the timeliness of evaluation, and evaluation items. CONCLUSIONS: A competency-based operative skill assessment can be incorporated into surgical training, assessment, and certification. The time has come to develop a systematic approach to this issue as a means of demonstrating professional standards worthy of the public trust.

Medical Students' Progress in Detecting and Interpreting Visual and Auditory Clinical Findings.

Phenomenon: Detection of visual and auditory clinical findings is part of medical students' core clinical performance abilities that a medical education curriculum should teach, assess, and remediate. However, there is a limited understanding of how students develop these skills. While training physical exam technical skills has received significant attention and emphasis, teaching and assessing medical students' ability to detect and interpret visual and auditory clinical findings skills has been less systematic. Therefore, the purpose of this study is to investigate how medical students' visual and auditory clinical findings skills progress and develop over their four years of undergraduate medical education. This study will provide educators insights that can guide curriculum refinements that lead to improving students' abilities in this area. Approach: A computer-based progress exam was created to measure the longitudinal development of students' abilities to detect and interpret visual and auditory findings. After pilot testing, sixty test items were developed in collaboration with six clinical faculty members and two medical education researchers. The exam includes detection and description of ECG, x-ray, heart sounds, breath sounds, skin lesions, and movement findings. The exam was administered to students at the beginning of each training year since 2014. Additionally, the exam was administered to the Class of 2017 prior to their graduation. Measurement validity and reliability tests were conducted. Descriptive statistics and ANOVA were used to determine progress. Findings: More than 98% of students in four years of training completed the exam each year. The exam instrument had high reliabilities and demonstrated acceptable concurrent validity when compared with other academic performance data. Findings showed that students' visual and auditory clinical findings skills increased each training year until their fourth year. There was no performance improvement between incoming Year 4 students and graduating Year 4 students. While group means increased, class performance did not become more homogeneous across four years. Longitudinal data showed the same performance patterns as the cross-sectional data. Performance of the bottom quartile of graduating fourth-year students was not significantly higher than the performance of the top quartile of incoming first-year students who had not had formal medical training. Insights: A longitudinal study to follow learners' performance in detecting and interpreting visual and auditory clinical findings can provide meaningful insights regarding the effects of medical training programs on performance growth. The present study suggests that our medical curriculum is not effective in bringing all students to a higher level of performance in detecting and interpreting visual and auditory clinical findings. This study calls for further investigation how medical students can develop visual and auditory detection and interpretation skills in undergraduate medical education. There is a need for planned curriculum and assessment of medical students' skills in detecting and interpreting visual and auditory clinical findings.

The journeys of three ASPIRE winning medical schools toward excellence in student assessment.

Introduction: ASPIRE Excellence Awards in Student Assessment are offered to medical schools with innovative and comprehensive assessment programmes adjudged by international experts, using evidence-based criteria. The journeys of three ASPIRE-winning medical schools toward "assessment excellence" are presented. These schools include Aga Khan University Medical College (AKU-MC), Pakistan, Southern Illinois University School of Medicine (SIUSOM), USA, and University of Leeds School of Medicine, UK. Methods: The unfolding journeys highlighting achievements, innovations, and essential components of each assessment programme were compared to identify differences and commonalities. Results: Cultural contextual differences included developed-versus-developing country, east-west, type of regulatory bodies, and institutional-versus-national certifying/licensing examinations, which influence curricula and assessments. In all, 12 essential commonalities were found: alignment with institutional vision; sustained assessment leadership; stakeholder engagement; communication between curriculum and assessment; assessment-for-learning and feedback; longitudinal student profiling of outcome achievement; assessment rigor and robustness; 360° feedback from-and-to assessment; continuous enrichment through rigorous quality assurance; societal sensitivity; influencing others; and a "wow factor." Conclusions: Although the journeys of the three medical schools were undertaken in different cultural contexts, similar core components highlight strong foundations in student assessment. The journeys continue as assessment programmes remain dynamic and measurement science expands. This article may be helpful to other institutions pursuing excellence in assessment.

Competencies, milestones, and EPAs - Are those who ignore the past condemned to repeat it?

BACKGROUND: The idea of competency-based education sounds great on paper. Who wouldn't argue for a standardized set of performance-based assessments to assure competency in graduating students and residents? Even so, conceptual concerns have already been raised about this new system and there is yet no evidence to refute their veracity. AIMS: We argue that practical concerns deserve equal consideration, and present evidence strongly suggesting these concerns should be taken seriously. METHOD: Specifically, we share two historical examples that illustrate what happened in two disparate contexts (K-12 education and the Department of Defense [DOD]) when competency (or outcomes-based) assessment frameworks were implemented. We then examine how observation and assessment of clinical performance stands currently in medical schools and residencies, since these methodologies will be challenged to a greater degree by expansive lists of competencies and milestones. RESULTS/CONCLUSIONS: We conclude with suggestions as to a way forward, because clearly the assessment of competency and the ability to guarantee that graduates are ready for medical careers is of utmost importance. Hopefully the headlong rush to competencies, milestones, and core entrustable professional activities can be tempered before even more time, effort, frustration and resources are invested in an endeavor which history suggests will collapse under its own weight.

Twenty Questions game performance on medical school entrance predicts clinical performance.

CONTEXT: This study is based on the premise that the game of 'Twenty Questions' (TQ) tests the knowledge people acquire through their lives and how well they organise and store it so that they can effectively retrieve, combine and use it to address new life challenges. Therefore, performance on TQ may predict how effectively medical school applicants will organise and store knowledge they acquire during medical training to support their work as doctors. OBJECTIVES: This study was designed to determine whether TQ game performance on medical school entrance predicts performance on a clinical performance examination near graduation. METHODS: This prospective, longitudinal, observational study involved each medical student in one class playing a game of TQ on a non-medical topic during the first week of medical school. Near graduation, these students completed a 14-case clinical performance examination. Performance on the TQ task was compared with performance on the clinical performance examination. RESULTS: The 24 students who exhibited a logical approach to the TQ task performed better on all senior clinical performance examination measures than did the 26 students who exhibited a random approach. Approach to the task was a better predictor of senior examination diagnosis justification performance than was the Medical College Admission Test (MCAT) Biological Science Test score and accounts for a substantial amount of score variation not attributable to a co-relationship with MCAT Biological Science Test performance. CONCLUSIONS: Approach to the TQ task appears to be one reasonable indicator of how students process and store knowledge acquired in their everyday lives and may be a useful predictor of how they will process the knowledge acquired during medical training. The TQ task can be fitted into one slot of a mini medical interview.

Building on Strengths: An Affirmational and Systems-Level Approach to Revisiting Character in Medicine.

ABSTRACT: Although U.S. medical education has continued to place increased emphasis on defining competency standards and ensuring accountability to the public, health care inequities have persisted, several basic health outcomes have worsened, public trust in the health care system has eroded, and moral distress, burnout, and attrition among practicing physicians have escalated. These opposing trends beg the question of how the "good doctor" concept may be strengthened. In this perspective, the authors argue that revisiting the construct of physician character from an affirmational perspective could meaningfully improve medical education's impact on overall health by more holistically conceptualizing what-and who-a good doctor is. The authors introduce positive psychology's framework of character strengths, probe the distinction between character strengths and medical professionalism, and summarize the role of character strengths in promoting physician engagement and well-being in health care work. They contend that a systems-level approach to cultivating character strengths will foster physician moral agency and well-being and, by extension, transformational change in health care. Consistent with best practice in modern character education, the authors propose that institutions mindfully cultivate moral community among all stakeholders (students, faculty, staff, postgraduate trainees, and patients) and that moral community interaction centers on each member's personal aspirations with respect to living a good life, guided by the character strengths framework and informed by patient perspectives.

Biomedical knowledge, clinical cognition and diagnostic justification: a structural equation model.

CONTEXT: The process whereby medical students employ integrated analytic and non-analytic diagnostic strategies is not fully understood. Analysing academic performance data could provide a perspective complementary to that of laboratory experiments when investigating the nature of diagnostic strategy. This study examined the performance data of medical students in an integrated curriculum to determine the relative contributions of biomedical knowledge and clinical pattern recognition to diagnostic strategy. METHODS: Structural equation modelling was used to examine the relationship between biomedical knowledge and clinical cognition (clinical information gathering and interpretation) assessed in Years 1 and 2 of medical school and their relative contributions to diagnostic justification assessed at the beginning of Year 4. Modelling was applied to the academic performance data of 133 medical students who received their md degrees in 2011 and 2012. RESULTS: The model satisfactorily fit the data. The correlation between biomedical knowledge and clinical cognition was low-moderate (0.26). The paths between these two constructs and diagnostic justification were moderate and slightly favoured biomedical knowledge (0.47 and 0.40 for biomedical knowledge and clinical cognition, respectively). CONCLUSIONS: The findings suggest that within the first 2 years of medical school, students possessed separate, but complementary, cognitive tools, comprising biomedical knowledge and clinical pattern recognition, which contributed to an integrated diagnostic strategy at the beginning of Year 4. Assessing diagnostic justification, which requires students to make their thinking explicit, may promote the integration of analytic and non-analytic processing into diagnostic strategy.

Patient safety education: what was, what is, and what will be?

Patient safety is an important topic that has been receiving more attention in the current health care climate. Patient safety as a curriculum topic in medical schools has only become apparent in the late 1990 s, and much more needs to be done. This article summarizes patient safety curricular content as it occurred (or did not occur) in medical education circles in the past (pre-1990 s), and present. It also makes some recommendations for the future of medical education curricula in the area of patient safety, using a framework for the development of expertise using the Dreyfus educational model.

The case of the entitled resident: a composite case study of a resident performance problem syndrome with interdisciplinary commentary.

BACKGROUND: Residents with performance problems create substantial burden on programs and institutions. Understanding the nature and quality of performance problems can help in learning to address performance problems. AIM: We sought to illuminate the effects of resident performance problems and the potential solutions for those problems from the perspectives of people with various roles in health care. METHODS: We created a composite portrait from several residents who demonstrated a cluster of common performance characteristics and whose chronic or serious maladaptive behavior and response to situations created problems for themselves, for their clinical colleagues, and for faculty of their residency program. The composite was derived from in-depth interviews of program directors and review of resident records. We solicited practitioners from multiple fields to respond to the portrait by answering a series of questions about severity, prognosis, and how and whether one could reliably remediate a person with these performance characteristics. We present their perspectives in a manner borrowed from the New England Journal of Medicine's "Case Records of the Massachusetts General Hospital." RESULTS: We created a composite portrait of a resident whose behavior suggested he felt entitled to benefits his peers were not entitled to. Experts reflecting on his behavior varied in their opinion about the effect the resident would have on the health care system. They suggested approaches to remediation that required substantial time and effort from the faculty. CONCLUSION: Programs must balance the needs of individual residents to adjust their behaviors with the needs of the health care system and other people within it.

The efficacy of a targeted remediation process for students who fail standardized patient examinations.

BACKGROUND: Current remediation strategies for students failing standardized patient examinations represent poorly targeted approaches since the specific nature of clinical performance weaknesses has not been defined. PURPOSE: The purpose is to determine the impact of a specifically targeted clinical performance course required of students who failed a clinical performance examination. METHODS: A month-long clinical performance course, targeted to treat specific types of clinical performance deficiencies, was designed to remediate students failing standardized patient examinations in 2007 (n=8) and 2008 (n=5). Participating students were assessed on pre- and postperformance measures, including multiple-choice tests that measured diagnostic pattern recognition and clinical data interpretation and clinical performance measures using standardized clinical encounters. Comparisons between average pre- and postintervention performance scores were computed using paired sample t tests. Results were adjusted for regression toward the mean. RESULTS: In both 2007 and 2008, the mean preintervention clinical data interpretation and standardized patient examination scores were below the criterion referenced passing standard set for the clinical competency exam. In both years the mean postintervention scores for the participants were above the passing standard for these two examinations. Pre- and postintervention differences were statistically significant in both cases. CONCLUSIONS: This study provides insight into the reasons that students fail clinical performance examinations and elucidates one method by which such students may be successfully remediated.

Designing a patient safety undergraduate medical curriculum: the Telluride Interdisciplinary Roundtable experience.

PURPOSE: Patient safety has emerged as a global concern in the provision of quality health care, and yet, to date, few medical schools have created and/or implemented patient safety curricula. The purpose of this article is to introduce readers to one model of a patient safety undergraduate medical curriculum, as designed by a group of experts attending an annual interdisciplinary roundtable assembled for this purpose. SUMMARY: The Annual Telluride Interdisciplinary Roundtable met in 2005 and 2006 to design what it considered to be a comprehensive patient safety curriculum for medical students. Invited members included stakeholders from a variety of fields, including health care providers, senior health care administration, students, residents, patient advocacy leaders, and curriculum development/assessment experts. The group developed a list of general curricular principles, followed by 11 specific elements felt to be essential to an effective patient safety curriculum for undergraduate medical education students. It also identified a number of challenges to implementing such a curriculum. CONCLUSIONS: A patient safety curriculum, developed by a group of experts for an undergraduate medical education population, was successfully developed over a two-year period of time. Future meetings of the Telluride Roundtable group have centered on evaluation and refinement of these curricular elements as pilots occur in a number of medical schools, and new curricular ideas continue to be developed. Continued interprofessional dialogue and collaborative research will enable the development and implementation of a standardized longitudinal patient safety student curriculum.

Supervisor and self-ratings of graduates from a medical school with a problem-based learning and standard curriculum track.

BACKGROUND: Medical schools recognize the importance of following up on graduates to evaluate performance during residency. Such performance is of particular interest for schools with problem-based learning (PBL) and conventional curriculum tracks. PURPOSES: Our purposes were to determine any differences in ratings of resident performance in 3 competency areas compared across two different curriculum tracks, 1st and 3rd postgraduate years, and self versus supervisor. METHODS: Performance ratings by residents and their supervisors for the 9 years during which 2 curriculum tracks were operative were analyzed using t tests. Effect sizes for statistically significant results were calculated. RESULTS: Several comparisons found differences, although effect sizes were almost uniformly small. The exception was the rating of self-directed learning habits, in which graduates from the problem-based track rated themselves higher than graduates from the standard curriculum (both Years 1 and 3) and higher than their supervisors in Year 3. CONCLUSIONS: Ratings by supervisors for both groups and both rating times are above average in the rating scale. Supervisors differentiated between the curricula tracks in postgraduate year 3 (PGY-3), rating Standard Curriculum graduates higher than PBL graduates in 5 of 6 noncognitive items and in two of three general ratings. Supervisor ratings increased between PGY-1 and PGY-3 for Standard Curriculum graduates in 9 of the competencies, whereas there were no changes for PBL graduates. In conclusion, performance differences and effect sizes are minimal, and it is possible to conclude that graduates from the 2 curricular tracks achieve at similar levels during their residency programs.

Getting Real: Aligning the Learning Needs of Clerkship Students With the Current Clinical Environment.

The authors present follow-up to a prior publication, which proposed a new model for third-year clerkships. The new model was created to address deficiencies in the clinical year and to rectify a recognized mismatch between students' learning needs and the realities of today's clinical settings. The new curricular model was implemented at Southern Illinois University School of Medicine in academic year 2016-2017. Guiding principles were developed. These were to more deeply engage students in experiential learning through clinical immersion; to pair individual faculty with individual students over longer periods of time so real trust could be developed; to provide students with longitudinal clinical reasoning education under controlled instructional conditions; to simplify goals and objectives for the core clerkships and align them with student learning needs; and to provide students with individualized activities to help them explore areas of interest, choose their specialty, and improve areas of clinical weakness before the fourth year. The authors discuss reactions by faculty and students to the new curriculum, which were mostly positive, as well as several outcomes. Students showed very different attitudes toward what they defined as success in the clerkship year, reflective of their deeper immersion. Students spent more time working in clinical settings and performed more procedures. Performance on Step 2 Clinical Knowledge and Clinical Skills was unchanged from traditional clerkship years. The 2015 article called for rethinking the third-year clerkships. The authors have shown that such change is possible, and the new curriculum can be implemented with successful early outcomes.

A sampling strategy for promoting and assessing medical student retention of physical examination skills.

BACKGROUND: Skill acquisition and maintenance requires spaced deliberate practice. Assessing medical students' physical examination performance ability is resource intensive. The authors assessed the nature and size of physical examination performance samples necessary to accurately estimate total physical examination skill. METHOD: Physical examination assessment data were analyzed from second year students at the University of Illinois College of Medicine at Chicago in 2002, 2003, and 2004 (N = 548). Scores on subgroups of physical exam maneuvers were compared with scores on the total physical exam, to identify sound predictors of total test performance. RESULTS: Five exam subcomponents were sufficiently correlated to overall test performance and provided adequate sensitivity and specificity to serve as a means to prompt continued student review and rehearsal of physical examination technical skills. CONCLUSIONS: Selection and administration of samples of the total physical exam provide a resource-saving approach for promoting and estimating overall physical examination skills retention.

ASPIRE-ing to Excellence at SIUSOM.

This article was migrated. The article was marked as recommended. This Personal View article describes the experience of Southern Illinois University School of Medicine (SIUSOM) with the AMEE School Programme for International Recognition of Excellence in Education (ASPIRE) awards program. Institutional leaders considering applying may need something more than the program description to take the plunge. We use narrative to present our reasons for applying, how the application and review process went for us, and the benefits of getting involved. By sharing our story, we hope to motivate other educators who believe in their school's educational excellence to visualize themselves as applicants and take action.

See one, do one, teach one--exploring the core teaching beliefs of medical school faculty.

This paper explores the core teaching beliefs of medical school faculty and establishes whether these beliefs differ among basic science, clinical, and instruction specialist faculty. One hundred and twenty-five medical school teachers who were members of professional organizations dedicated to the improvement of medical school teaching completed a Q-sort of 56 statements reflecting their core teaching beliefs. The statements described beliefs about motivation, knowledge and skill acquisition, retention, feedback, transfer, teacher characteristics, and teaching strategies. Q-sorts were completed by 37 basic scientists (30% of respondents), 59 clinicians (47%) and 29 instruction specialists (23%) working in medical schools. Fifty-two participants were classroom teachers (42%), 66 were classroom and clinical teachers (53%), and seven reported that they do not teach (6%). The Q-sort results indicate how medical school faculty members differ in their core beliefs about teaching and learning. Thirty-two respondents (26%) focused on the student as a person first. Eight (6%) were content oriented. Thirty-four (27%) were performance oriented; their focus was on having students learn and apply knowledge and skills to accomplish clinical tasks. Fifty-one respondents (41%) were found to have a blend of these viewpoints. Respondents' type of training or type of teaching did not provide a reliable indication of core teaching beliefs classification.

Forecasting residents' performance--partly cloudy.

The authors offer a practical guide for improving the appraisal of a resident's performance. They identify six major factors that compromise the process of observing, measuring, and characterizing a resident's current performance, forecasting future performance, and making decisions about the resident's progress. Factors that compromise any of these steps lead to individual and collective uncertainty and decrease faculty confidence when making decisions on a resident's progress. The six factors, addressed in order of importance, are inaccuracies due to (1) incomplete sampling of performance, (2) rater memory constraints, (3) hidden performance deficits of the resident, (4) lack of performance benchmarks, (5) faculty members' hesitancy to act on negative performance information, and (6) systematic rater error. The description of each factor is followed by a number of specific suggestions on what residency programs can do to eliminate or minimize the impact of these factors. While this article is couched in the context of the performance evaluation of residents, everything included pertains to measuring and appraising medical students' and practicing physicians' clinical performance as well.

Improving resident performance assessment data: numeric precision and narrative specificity.

PURPOSE: To evaluate the use of a systems approach for diagnosing performance assessment problems in surgery residencies, and intervene to improve the numeric precision of global rating scores and the behavioral specificity of narrative comments. METHOD: Faculty and residents at two surgery programs participated in parallel before- and-after trials. During the baseline year, quality assurance data were gathered and problems were identified. During two subsequent intervention years, an educational specialist at each program intervened with an organizational change strategy to improve information feedback loops. Three quality-assurance measures were analyzed: (1) percentage return rate of forms, (2) generalizability coefficients and 95% confidence intervals of scores, and (3) percentage of forms with behaviorally specific narrative comments. RESULTS: Median return rates of forms increased significantly from baseline to intervention Year 1 at Site A (71% to 100%) and Site B (75% to 100%), and then remained stable during Year 2. Generalizability coefficients increased between baseline and intervention Year 1 at Site A (0.65 to 0.85) and Site B (0.58 to 0.79), and then remained stable. The 95% confidence interval around resident mean scores improved at Site A from baseline to intervention Year 1 (0.78 to 0.58) and then remained stable; at Site B, it remained constant throughout (0.55 to 0.56). The median percentage of forms with behaviorally specific narrative comments at Site A increased significantly from baseline to intervention Years 1 and 2 (50%, 57%, 82%); at Site B, the percentage increased significantly in intervention Year 1, and then remained constant (50%, 60%, 67%). CONCLUSIONS: Diagnosing performance assessment system problems and improving information feedback loops improved the quality of resident performance assessment data at both programs.