Educator's blueprint: A how-to guide for developing high-quality multiple-choice questions.

Multiple-choice questions are commonly used for assessing learners' knowledge, as part of educational programs and scholarly endeavors. To ensure that questions accurately assess the learners and provide meaningful data, it is important to understand best practices in multiple-choice question design. This Educator's Blueprint paper provides 10 strategies for developing high-quality multiple-choice questions. These strategies include determining the purpose, objectives, and scope of the question; assembling a writing team; writing succinctly; asking questions that assess knowledge and comprehension rather than test-taking ability; ensuring consistent and independent answer choices; using plausible foils; avoiding grouped options; selecting the ideal response number and order; writing high-quality explanations; and gathering validity evidence before and evaluating the questions after use.

Resident Physician Interactions and Engagement With Written Assessments of Performance.

BACKGROUND AND OBJECTIVES: Despite the requirements of the Accreditation Council for Graduate Medical Education (ACGME) to provide feedback, assessments are often not meeting the needs of resident learners. The objective of this study was to explore residents' approach to reviewing, interpreting, and incorporating the feedback provided in written faculty assessments. METHODS: We conducted semistructured interviews with 14 family medicine residents. We used line-by-line iterative coding of the transcripts through the constant comparative method to identify themes and reach a consensus. RESULTS: The study revealed the following themes: (1) residents value the narrative portion of assessments over numerical ratings, (2) performance reflection and reaction are part of the feedback process, (3) residents had difficulty incorporating formal assessments as many did not provide actionable feedback. CONCLUSIONS: Residents reported that narrative feedback gives more insight to performance and leads to actionable changes in behaviors. Programs should consider education for both faculty and residents on the usefulness, importance, and purpose of the ACGME Milestones in order to accurately determine resident competency and provide a summative assessment. Until the purpose of the ACGME Milestones is realized and utilized, it should be noted that the comment portion of evaluations will likely be the focus of the resident's interaction with their assessments.

The test developer's dilemma: Evaluating the balance of feasibility and empiric performance of test development techniques for repeated written assessments.

PURPOSE: Written assessments face challenges when administered repeatedly, including resource-intensive item development and the potential for performance improvement secondary to item recall as opposed to understanding. This study examines the efficacy of three-item development techniques in addressing these challenges. METHODS: Learners at five training programs completed two 60-item repeated assessments. Items from the first test were randomized to one of three treatments for the second assessment: (1) Verbatim repetition, (2) Isomorphic changes, or (3) Total revisions. Primary outcomes were the stability of item psychometrics across test versions and evidence of item recall influencing performance as measured by the rate of items answered correctly and then incorrectly (correct-to-incorrect rate), which suggests guessing. RESULTS: Forty-six learners completed both tests. Item psychometrics were comparable across test versions. Correct-to-incorrect rates differed significantly between groups with the highest guessing rate (lowest recall effect) in the Total Revision group (0.15) and the lowest guessing rate (highest recall effect) in the Verbatim group (0.05), p = 0.01. CONCLUSIONS: Isomorphic and total revisions demonstrated superior performance in mitigating the effect of recall on repeated assessments. Given the high costs of total item revisions, there is promise in exploring isomorphic items as an efficient and effective approach to repeated written assessments.[Box: see text].

Preparing Effective Narrative Evaluations for the Medical School Performance Evaluation (MSPE).

Introduction: In 2016, the AAMC Medical School Performance Evaluation (MSPE) Task Force issued recommendations to standardize the MSPE but did not address the quality of the written narratives in that document. Narrative evaluations are hampered by code words, polite rhetoric, and bias to the detriment of students. To address this, the AAMC's Group on Student Affairs and Group on Educational Affairs convened an expert group to consider the state of narratives in the MSPE and develop resources to improve their quality. Methods: A series of interactive workshops was developed and presented at an AAMC webinar and national meetings. A presentation outlining challenges and possible approaches to improvement was followed with large-group discussion and/or small-group breakout activity to analyze and improve upon sample clinical comments and create summary clerkship paragraphs. The initial webinar used polling questions and free-text prompts to gather feedback for future workshops. Anonymous survey responses were collected at the end of each subsequent workshop to determine perceived effectiveness and potential utility at participants' institutions. Results: Over 680 administrators, faculty, and staff participated in the webinar or in one of four national-level workshops. Respondents agreed that the modules would be useful in faculty development and wanted to replicate their learning at their own institutions for overall better impact on the quality of MSPE narratives. Discussion: This resource addresses an important gap in the medical education literature. A variety of stakeholders affirmed that these workshops have value in training writers to improve their narrative comments for the MSPE.

Graduate medical education tool kit for the Supplemental Offer and Acceptance Program: Navigating the National Resident Matching Program supplemental offer assistance program.

More residency training programs are involved in the National Resident Matching Program (NRMP) Supplemental Offer and Acceptance Program (SOAP) than in the past, which is unanticipated by many. Without an action plan in place, the SOAP process can be chaotic and could result in offers to incompletely vetted residency applicants and unfilled residency slots. Faced with this challenge, it is important that programs have a unified, clear process and action plan for the SOAP. Our instution has created a SOAP tool kit that documents our strategies, scripts, and timelines in preparation of SOAP. The success of our approach has been gauged by the positive response from the key stakeholders who found this easy to use without significant advanced training. Faculty and staff noted significant efficiencies and improvements in the process when using the SOAP tool kit. Applicants indicated that the process was more organized than other residency programs they interacted with during the SOAP week. We anticipate that the use of the documents in the SOAP tool kit will help other training programs improve their performance during the SOAP process.

The Council of Emergency Medicine Residency Directors Academy for Scholarship Coaching Program: Addressing the Needs of Academic Emergency Medicine Educators.

INTRODUCTION: Didactic lectures remain fundamental in academic medicine; however, many faculty physicians do not receive formal training in instructional delivery. In order to design a program to instill and enhance lecture skills in academic emergency medicine (EM) physicians we must first understand the gap between the current and ideal states. METHODS: In 2012 the Council of Emergency Medicine Residency Directors (CORD) Academy for Scholarship designed a novel coaching program to improve teaching skills and foster career development for medical educators based on literature review and known teaching observation programs. In order to inform the refinement of the program, we performed a needs assessment of participants. Participants' needs and prior teaching experiences were gathered from self-reflection forms completed prior to engaging in the coaching program. Two independent reviewers qualitatively analyzed data using a thematic approach. RESULTS: We analyzed data from 12 self-reflection forms. Thematic saturation was reached after nine forms. Overall inter-rater agreement was 91.5%. We categorized emerging themes into three domains: participant strengths and weaknesses; prior feedback with attempts to improve; and areas of desired mentorship. Several overlapping themes and subthemes emerged including factors pertaining to the lecturer, the audience/learner, and the content/delivery. CONCLUSION: This study identified several areas of need from EM educators regarding lecture skills. These results may inform faculty development efforts in this area. The authors employed a three-phase, novel, national coaching program to meet these needs.

Not another boring lecture: engaging learners with active learning techniques.

BACKGROUND: Core content in Emergency Medicine Residency Programs is traditionally covered in didactic sessions, despite evidence suggesting that learners do not retain a significant portion of what is taught during lectures. DISCUSSION: We describe techniques that medical educators can use when leading teaching sessions to foster engagement and encourage self-directed learning, based on current literature and evidence about learning. CONCLUSIONS: When these techniques are incorporated, sessions can be effective in delivering core knowledge, contextualizing content, and explaining difficult concepts, leading to increased learning.

A Comparison between Emergency Medicine Residency Training Programs in the United States and Saudi Arabia from the Residents' Perception.

Objectives. This study was designed to compare the trainees' perception of emergency medicine (EM) training in the United States (US) and Saudi Arabia (SA) and to identify residents' levels of confidence and points of satisfaction in education, procedural skills, and work environment. Method. An IRB-exempt anonymous web-based survey was distributed to five EM residency training programs in the USA and three residency regions in SA. Results. 342 residents were polled with a 20% response rate (16.8% USA and 25.8% SA). The Saudi residents responded less positively to the questions about preparation for their boards' examinations, access to multiple educational resources, and weekly academic activities. The Saudi trainees felt less competent in less common procedures than US trainees. American trainees also more strongly agree that they have more faculty interest in their education compared to the Saudi trainees. The Saudi residents see more patients per hour compared to their US peers. Conclusion. These findings may be due to the differences in training techniques including less formal didactics and simulation experience in SA and more duty hour regulations in the USA.

The development of the emergency medicine milestones.

The Accreditation Council for Graduate Medical Education (ACGME) has outlined its "Next Accreditation System" (NAS) that will focus on resident and residency outcome measurements. Emergency medicine (EM) is one of seven specialties that will implement the NAS beginning July 2013. All other specialties will follow in July 2014. A key component of the NAS is the development of assessable milestones, which are explicit accomplishments or behaviors that occur during the process of residency education. Milestones describe competencies more specifically and identify specialty-specific knowledge, skills, attitudes, and behaviors (KSABs) that can be used as outcome measures within the general competencies. The ACGME and the American Board of Emergency Medicine (ABEM) convened an EM milestone working group to develop the EM milestones. This article describes the development, use within the NAS, and challenges of the EM milestones.

Duty hours in emergency medicine: balancing patient safety, resident wellness, and the resident training experience: a consensus response to the 2008 institute of medicine resident duty hours recommendations.

Representatives of emergency medicine (EM) were asked to develop a consensus report that provided a review of the past and potential future effects of duty hour requirements for EM residency training. In addition to the restrictions made in 2003 by the Accreditation Council for Graduate Medical Education (ACGME), the potential effects of the 2008 Institute of Medicine (IOM) report on resident duty hours were postulated. The elements highlighted include patient safety, resident wellness, and the resident training experience. Many of the changes and recommendations did not affect EM as significantly as other specialties. Current training standards in EM have already emphasized patient safety by requiring continuous on-site supervision of residents. Resident fatigue has been addressed with restrictions of shift lengths and limitation of consecutive days worked. One recommendation from the IOM was a required 5-hour rest period for residents on call. Emergency department (ED) patient safety becomes an important concern with the decrease in the availability and in the patient load of a resident consultant that may result from this recommendation. Of greater concern is the already observed slower throughput time for admitted patients waiting for resident care, which will increase ED crowding and decrease patient safety in academic institutions. A balance between being overly prescriptive with duty hour restrictions and trying to improve resident wellness was recommended. Discussion is included regarding the appropriate length of EM training programs if clinical experiences were limited by new duty hour regulations. Finally, this report presents a review of the financing issues associated with any changes.

Duty hours in emergency medicine: balancing patient safety, resident wellness, and the resident training experience: a consensus response to the 2008 Institute of Medicine resident duty hours recommendations.

BACKGROUND: Representatives of emergency medicine (EM) were asked to develop a consensus report that provided a review of the past and potential future effects of duty hour requirements for EM residency training. In addition to the restrictions made in 2003 by the Accreditation Council for Graduate Medical Education, the potential effects of the 2008 Institute of Medicine (IOM) report on resident duty hours were postulated. DISCUSSION: The elements highlighted include patient safety, resident wellness, and the resident training experience. Many of the changes and recommendations did not affect EM as significantly as other specialties. Current training standards in EM have already emphasized patient safety by requiring continuous onsite supervision of residents. Resident fatigue has been addressed with restrictions of shift lengths and limitation of consecutive days worked. CONCLUSION: One recommendation from the IOM was a required 5-h rest period for residents on call. Emergency department (ED) patient safety becomes an important concern with the decrease in the availability and in the patient load of a resident consultant that may result from this recommendation. Of greater concern is the already observed slower throughput time for admitted patients waiting for resident care, which will increase ED crowding and decrease patient safety in academic institutions. A balance between being overly prescriptive with duty hour restrictions and trying to improve resident wellness was recommended. Discussion is included regarding the appropriate length of EM training programs if clinical experiences were limited by new duty hour regulations. Finally, this report presents a review of the financing issues associated with any changes.

Importance of residency program web sites to emergency medicine applicants.

Emergency Medicine (EM) residency program web sites are an important tool that programs use to attract applicants. However, there are only a few studies examining the aspects of a program's web site that are most important to EM applicants. We conducted a cross-sectional study of 142 prospective residency applicants interviewing for an EM position at one of three EM residency programs for the 2003 match. The survey demonstrated that almost all applicants researched EM programs online. The majority (71%) identified geographic location as the most important factor in applying to a specific program. Approximately 40% considered an easily navigated web site as very/moderately important to their application decision-making process. Rotation schedule was also important in applicant decision-making. The Internet is a significant source of information to the majority of applicants in EM. Online information from programs' web sites, although not as significant as geography, influences an applicant's choice of where to apply for a residency position. An easily navigated, complete web site may improve the recruitment of candidates to EM residency programs.

Competence of new emergency medicine residents in the performance of lumbar punctures.

BACKGROUND: Medical students are taught some procedural skills during medical school, but there is no uniform set of procedures that all students learn before residency. OBJECTIVE: To determine the level of competence in the performance of a lumbar puncture (LP) by new postgraduate year 1 (PGY1) emergency medicine (EM) residents. METHODS: An observational study was conducted at three EM residencies with 42 PGY1 residents who recently graduated from 26 various medical schools. The LP procedure was divided into 26 major and 44 minor steps to create a scoring protocol. The model, procedure, and scoring protocol were validated by experienced emergency physicians. Subjects performed the procedure without interruption or feedback on an LP training model using a standard LP kit. A step was scored as "performed correctly" if two of the three evaluators concurred. Pre- and poststudy questionnaires assessed subjects' prior instruction and clinical experience with LP, self-confidence, sense of relevance, motivation, and fatigue. RESULTS: Subjects completed an average of 14.8 (57%; 95% confidence interval [95% CI] = 53% to 61%) of the major steps (range: 4-26) and 19.1 (43%; 95% CI = 42% to 45%) of the minor steps (range: 7-28) in 14.3 minutes (range: 3-22). Sixty-nine percent failed to obtain cerebrospinal fluid from the model. Subjects' levels of confidence changed slightly on a five-point scale from 2.8 ("little-to-some") before the test to 2.5 after the test. Eighty-three percent of the subjects previously performed LPs on patients during medical school (average attempts = 2.2; range: 0-10), but only 40% of those who did so were supervised by an attending during their first attempt. CONCLUSIONS: In the cohort studied, new PGY1 EM residents had not attained competence in performing LPs from training in medical school. Most new PGY1 residents probably require training, practice, and close, direct supervision of this procedure by attending physicians until the residents demonstrate competent performance.