Building a student learning-focused assessment and grading system in dental school: One school's experience.

PURPOSE/OBJECTIVES: As health professions education moves toward competency-based education, there has been increased focus on the structure of assessment systems that support student competency development and learning. This has been buoyed by a growing body of research supporting assessment for learning processes to promote student growth and learning rather than relying on assessment systems primarily to measure performance. This paper presents the rationale and evidence for moving to an assessment for learning system and the results of a quasi-experimental interrupted time series study using data from 2015 to 2022 to evaluate the impacts of these changes. METHODS: Columbia University College of Dental Medicine faculty voted to implement assessment for learning system changes beginning in 2017 with the graduating class of 2021. These changes included moving from using a grading system for didactic courses with Honors, Pass, and Fail as available grades to a grading system with only Pass and Fail as available grades, as well as creating synthesis and assessment weeks, weekly problem sets, post-exam review sessions, exam remediation opportunities, and formative progress exams throughout the curriculum. The revised assessment and grading system changes were communicated to residency program directors, and programmatic competency data about student performance across the curriculum were shared with programs in Dean's Letters. RESULTS: Once assessment system changes were implemented, it was found that student exam failure rates were lower, course exam scores were the same or higher, and performance on board exams improved compared to the national average. Students reported positive perceptions with regard to well-being and learning climate that they associated with the adoption of Pass/Fail grading. Match outcomes, including student satisfaction and program director ratings, have remained consistently positive. CONCLUSION: As dental educators, our goal is to nurture students to become life-long learners. Adopting a grading structure that is Pass/Fail and an assessment system that fosters learning allows students to shape learning practices that favor long-term retention and application of information, also enhancing the learning environment and student well-being. These system changes may also facilitate the inclusion and support of students whose backgrounds are underrepresented in dentistry.

Twelve tips on guiding preparation for both high-stakes exams and long-term learning.

High-stakes exams including admissions, licensing, and maintenance of certification examinations are commonplace in health professions education. Although exam scores and performance can often serve gate-keeping purposes, the broader goal of health professions education is to foster deep, self-directed, meaningful, motivated learning. Establishing strong support structures that emphasize deep learning and understanding rather than exam scores can be beneficial to preparing learners who have the knowledge base to be excellent practitioners. This article offers guidance that can be used by academic support centres, medical educators, learning specialists, and faculty advisors, or even test-takers, to help learners to balance score achievement and knowledge development, while simultaneously cultivating more efficient and motivated studying and increasingly self-regulated learning. This series of tips details considerations for building academic success supports, fostering a growth mindset, planning efficient and effective studying efforts, utilizing test-enhanced learning strategies, exam-taking skills practice, and other support structures that can help strengthen learning experiences overall.

Twelve tips for embedding assessment for and as learning practices in a programmatic assessment system.

Programmatic assessment supports the evolution from assessment of learning to fostering assessment for learning and as learning practices. A well-designed programmatic assessment system aligns educational objectives, learning opportunities, and assessments with the goals of supporting student learning, making decisions about student competence and promotion decisions, and supporting curriculum evaluation. We present evidence-based guidance for implementing assessment for and as learning practices in the pre-clinical knowledge assessment system to help students learn, synthesize, master and retain content for the long-term so that they can apply knowledge to patient care. Practical tips are in the domains of culture and motivation of assessment, including how an honour code and competency-based grading system can support an assessment system to develop student self-regulated learning and professional identity, curricular assessment structure, such as how and when to utilize low-stakes and cumulative assessment to drive learning, exam and question structure, including what authentic question and exam types can best facilitate learning, and assessment follow-up and review considerations, such exam retake processes to support learning, and academic success structures. A culture change is likely necessary for administrators, faculty members, and students to embrace assessment as most importantly a learning tool for students and programs.

Toward a Bias-Free and Inclusive Medical Curriculum: Development and Implementation of Student-Initiated Guidelines and Monitoring Mechanisms at One Institution.

As research and attention on implicit bias and inclusiveness in medical school is expanding, institutions need mechanisms for recognizing, reporting, and addressing instances of implicit bias and lack of inclusiveness in medical school curricular structures. These instances can come as a result of a lack of both awareness and communication around these sensitive issues. To identify and address cases of implicit bias in the medical school curriculum, a student-led initiative at Columbia University Vagelos College of Physicians and Surgeons (VP&S) developed guidelines and a bias-reporting process for educators and students. The guidelines, co-created by students and faculty, help educators identify and address implicit bias in the curriculum. Furthermore, to allow for continued development of the curriculum and the guidelines themselves, the group adapted an existing learning environment reporting and review process to identify and address instances of implicit bias. In the first year since their implementation, these tools have already had an impact on the learning climate at VP&S. They have led to enhanced identification of implicit bias in the curriculum and changes in instructional materials. The courage and inspiration of the students and the initial investment and commitment from the administration and faculty were crucial to this rapid effect. The authors present an approach and resources from which other institutions can learn, with the goal of reducing implicit bias and improving inclusiveness throughout medical education. In the long run, the authors hope that these interventions will contribute to better preparing future providers to care for all patients equitably.

Identifying and Supporting Students to Prevent USMLE Step 1 Failures When Testing Follows Clerkships: Insights From 9 Schools.

Several schools have moved the United States Medical Licensing Examination Step 1 exam after core clerkships, and others are considering this change. Delaying Step 1 may improve Step 1 performance and lower Step 1 failure rates. Schools considering moving Step 1 are particularly concerned about late identification of struggling students and late Step failures, which can be particularly problematic due to reduced time to remediate and accumulated debt if remediation is ultimately unsuccessful. In the literature published to date, little attention has been given to these students. In this article, authors from 9 medical schools with a postclerkship Step 1 exam share their experiences. The authors describe curricular policies, early warning and identification strategies, and interventions to enhance success for all students and struggling students in particular. Such learners can be identified by understanding challenges that place them "at risk" and by tracking performance outcomes, particularly on other standardized assessments. All learners can benefit from early coaching and advising, mechanisms to ensure early feedback on performance, commercial study tools, learning specialists or resources to enhance learning skills, and wellness programs. Some students may need intensive tutoring, neuropsychological testing and exam accommodations, board preparation courses, deceleration pathways, and options to postpone Step 1. In rare instances, a student may need a compassionate off-ramp from medical school. With the National Board of Medical Examiner's announcement that Step 1 scoring will change to pass/fail as early as January 2022, residency program directors might use failing Step 1 scores to screen out candidates. Institutions altering the timing of Step 1 can benefit from practical guidance by those who have made the change, to both prevent Step 1 failures and minimize adverse effects on those who fail.

Effects of Moving the United States Medical Licensing Examination Step 1 After Core Clerkships on Step 2 Clinical Knowledge Performance.

PURPOSE: To investigate the effect of a change in the United States Medical Licensing Examination Step 1 timing on Step 2 Clinical Knowledge (CK) scores, the effect of lag time on Step 2 CK performance, and the relationship of incoming Medical College Admission Test (MCAT) score to Step 2 CK performance pre and post change. METHOD: Four schools that moved Step 1 after core clerkships between academic years 2008-2009 and 2017-2018 were analyzed. Standard t tests were used to examine the change in Step 2 CK scores pre and post change. Tests of differences in proportions were used to evaluate whether Step 2 CK failure rates differed between curricular change groups. Linear regressions were used to examine the relationships between Step 2 CK performance, lag time and incoming MCAT score, and curricular change group. RESULTS: Step 2 CK performance did not change significantly (P = .20). Failure rates remained highly consistent (pre change: 1.83%; post change: 1.79%). The regression indicated that lag time had a significant effect on Step 2 CK performance, with scores declining with increasing lag time, with small but significant interaction effects between MCAT and Step 2 CK scores. Students with lower incoming MCAT scores tended to perform better on Step 2 CK when Step 1 was after clerkships. CONCLUSIONS: Moving Step 1 after core clerkships appears to have had no significant impact on Step 2 CK scores or failure rates, supporting the argument that such a change is noninferior to the traditional model. Students with lower MCAT scores benefit most from the change.

Investigating Group Differences in Examinees' Preparation for and Performance on the New MCAT Exam.

In 2015, the Medical College Admission Test (MCAT) was redesigned to better assess the concepts and reasoning skills students need to be ready for the medical school curriculum. During the new exam's design and rollout, careful attention was paid to the opportunities examinees had to learn the new content and their access to free and low-cost preparation resources. The design committee aimed to mitigate possible unintended effects of the redesign, specifically increasing historical mean group differences in MCAT scores for examinees from lower socioeconomic status (SES) backgrounds and races/ethnicities underrepresented in medicine compared with those from higher SES backgrounds and races/ethnicities not underrepresented in medicine.In this article, the authors describe the characteristics and scores of examinees who took the new MCAT exam in 2017 and compare those trends with historical ones from 2013, presenting evidence that the diversity and performance of examinees has remained stable even with the exam's redesign. They also describe the use of free and low-cost MCAT preparation resources and MCAT preparation courses for examinees from higher and lower SES backgrounds and who are enrolled in undergraduate institutions with more and fewer resources, showing that examinees from lower SES backgrounds and who attend institutions with fewer resources use many free and low-cost test preparation resources at lower rates than their peers. The authors conclude with a description of the next phase of this research: to gather qualitative and quantitative data about the preparation strategies, barriers, and needs of all examinees, but especially those from lower SES and underrepresented racial/ethnic backgrounds.

Challenges Associated With Moving the United States Medical Licensing Examination (USMLE) Step 1 to After the Core Clerkships and How to Approach Them.

An increasing number of medical schools have moved away from traditional 2 + 2 curricular structures toward curricula that intentionally integrate basic, clinical, and health systems science, with the goal of graduating physicians who consistently apply their foundational knowledge to clinical practice to improve the care of patients and populations. These curricular reforms often include a shortened preclerkship phase with earlier introduction of learners into clinical environments. This has led schools to reconsider the optimal timing of United States Medical Licensing Examination Step 1. A number of schools have shifted the exam to the period immediately after core clerkships. Although this shift can provide pedagogical advantages, there are potential challenges that must be anticipated and proactively addressed. As more institutions consider making this change, key educational leaders from five schools that repositioned the Step 1 exam after core clerkships share strategies for mitigating some of the potential challenges associated with this approach. The authors describe six possible challenges: lack of readiness without consolidation of basic science knowledge prior to clerkships; risk that weaker students will not be identified and provided academic support early; clerkship or clinical shelf exam performance weaknesses; extension of Step 1 study time; an increase in student anxiety about residency specialty choices; and/or a reduced time frame to take and pass board exams. These potential challenges may be addressed using three main strategies: effective communication with all stakeholders; curricular design and assessments that facilitate integration of basic and clinical sciences; and proactive student coaching and advising.

Moving the United States Medical Licensing Examination Step 1 After Core Clerkships: An Outcomes Analysis.

PURPOSE: Schools undergoing curricular reform are reconsidering the optimal timing of Step 1. This study provides a psychometric investigation of the impact on United States Medical Licensing Examination Step 1 scores of changing the timing of Step 1 from after completion of the basic science curricula to after core clerkships. METHOD: Data from four schools that recently moved the examination were analyzed in a pre-post format using examinee scores from three years before and after the change. The sample included scores from 2008 through 2016. Several confounders were addressed, including rising national scores and potential differences in cohort abilities using deviation scores and analysis of covariance (ANCOVA) controlling for Medical College Admission Test (MCAT) scores. A resampling procedure compared study schools' score changes versus similar schools' in the same time period. RESULTS: The ANCOVA indicated postchange Step 1 scores were higher compared with prechange (adjusted difference = 2.67; 95% confidence interval: 1.50-3.83, P < .001; effect size = 0.14) after adjusting for MCAT scores and rising national averages. The average score increase in study schools was larger than changes seen in similar schools. Failure rates also decreased from 2.87% (n = 48) pre change to 0.39% (n = 6) post change (P < .001). CONCLUSIONS: Results suggest moving Step 1 after core clerkships yielded a small increase in scores and a reduction in failure rates. Although these small increases are unlikely to represent meaningful knowledge gains, this demonstration of "noninferiority" may allow schools to implement significant curricular reforms.

Art as Sanctuary: A Four-Year Mixed-Methods Evaluation of a Visual Art Course Addressing Uncertainty Through Reflection.

PURPOSE: Engagement with visual art is a promising modality for addressing issues of uncertainty via reflective practice, one that is being increasingly used in health science education. An elective museum-based course for first-year medical students was created by two medical schools and led by an art educator experienced in medical education. The course, Observation and Uncertainty in Art and Medicine, sought to help students explore experiences of uncertainty and to develop reflective capacity through engagement with visual art. METHOD: The course was run and evaluated from 2014 to 2017, with 47 students participating over the 4 years, with 12 students enrolled per year. Before and after the course, students were given the Groningen Reflection Ability Scale (GRAS) for reflective ability, the Tolerance for Ambiguity scale for ambiguity, and Best Intentions Questionnaire for personal bias awareness, and 35 students (74%) completed all of the scales. Focus group interviews and narrative postcourse evaluations were conducted, coded, and thematically analyzed. RESULTS: Statistically significant improvement was found in GRAS scores. Qualitative themes included student enhancement of observational skills, awareness of the subjectivity and uncertainty of perception, exploration of multiple points of view, and recognition of the course as a place for restoration and connection to classmates. CONCLUSIONS: Incorporating visual art into medical education is an effective pedagogical method for addressing competencies central to training, including observation, reflection, and self-care.

Validity and reliability of a novel written examination to assess knowledge and clinical decision making skills of medical students on the surgery clerkship.

BACKGROUND: The Surgery Clerkship Clinical Skills Examination (CSE) is a novel written examination developed to assess the surgical knowledge, clinical decision making, communication skills, and professionalism of medical students on the surgery clerkship. This study was undertaken to determine its validity. METHODS: Data were prospectively collected from July 2011 through February 2013. Multivariate linear and logistic regression analyses were used to assess score trend; convergent validity with National Board of Medical Examiners surgery and medicine subject scores, United States Medical Licensing Examination Step 1 and Step 2 Clinical Knowledge scores, and evaluation of clinical reasoning and fund of knowledge; and the effect of clerkship order. Exam reliability was assessed using a modified Cronbach's α statistic. RESULTS: During the study period, 262 students completed the CSE, with a normal distribution of performance. United States Medical Licensing Examination Step 2 Clinical Knowledge score and end-of-clerkship evaluations of fund of knowledge and clinical reasoning predicted CSE score. Performance on the CSE was independent of clerkship order or prior clerkships. The modified Cronbach's α value for the exam was .67. CONCLUSIONS: The CSE is an objective, valid, reliable instrument for assessing students on the surgery clerkship, independent of clerkship order.

Sustaining an advisory dean program through continuous improvement and evaluation.

PURPOSE: In 2003, the advisory dean program at Columbia University College of Physicians and Surgeons was created to better connect students and faculty by supporting student academic progress and improving career advising. With the program in its eighth year, the authors were interested in identifying key factors in maintaining ongoing vitality and effectiveness. METHOD: In 2011, the authors conducted a reflective analysis to study the program, using available information from dean interviews, student surveys, meeting agendas, and program leader reflections, aided by the Bolman and Deal four-part framework for organizational functioning (structural, human resource, political, and symbolic). RESULTS: Structural factors included reframing program goals to match program activities, situating the program within broader academic advising and counseling resources, and increasing face time between entering students and their deans. Human resource factors included managing higher-than-expected turnover of deans with dean selection and orientation strategies that balance diversity and consistency, and providing ongoing training to promote continual professional growth. Political factors included balancing resources (e.g., money, administrative support) from the school and departments to help the deans protect and manage their time. Symbolic factors were related to leveraging the deans as symbols of institutional values and commitment to education (e.g., participation in the white coat ceremony), and being aware of "hidden meanings" associated with decisions within other frames (e.g., student-to-dean ratio). CONCLUSIONS: A variety of strategies across frames were used to maintain the program. This report can serve as a guide to program maintenance for other institutions.

Third-year medical student rotations in emergency medicine: a survey of current practices.

OBJECTIVES: Exposure to emergency medicine (EM) is a crucial aspect of medical student education, yet one that is historically absent from third-year medical student training. There are limited data describing the existing third-year rotations. The goal of this study is to identify the content and structure of current EM rotations specific to third-year students. METHODS: An institutional review board-approved survey of clerkship characteristics was designed by consensus opinion of clerkship directors (CDs). The survey was distributed to 32 CDs at institutions with known EM clerkships involving third-year students. RESULTS: Twenty-three (72%) CDs responded to the survey. Sixty-five percent have rotations designed specifically for third-year students, of which 33% are required clerkships. Twenty-seven percent of rotations have prerequisite rotations; 37% of rotations include shifts in the pediatric ED. Clinical time averages four 8-hour shifts per week for 4 weeks; all rotations include weekly didactic time specific to third-year students. A wide variety of textbooks are used; some programs employ simulation labs. Two-thirds of the rotations have a required write-up or presentation; 53% include a final exam. Student evaluations are written and verbal. Most rotations receive more support from the EM departments than from the medical schools for physical space, administrative needs, and faculty time. Among those surveyed, students from institutions requiring a third-year EM rotation have a higher rate of application to EM residencies. CONCLUSIONS: There is variability in the content and structure of existing third-year EM rotations, as well as in financial and administrative needs and support. These data can help to inform CDs and departments that are starting or modifying EM third-year rotations, as well as contribute to the development of curricula for such rotations.