A Multi-institutional Study of the Feasibility and Reliability of the Implementation of Constructed Response Exam Questions.

PROBLEM: Some medical schools have incorporated constructed response short answer questions (CR-SAQs) into their assessment toolkits. Although CR-SAQs carry benefits for medical students and educators, the faculty perception that the amount of time required to create and score CR-SAQs is not feasible and concerns about reliable scoring may impede the use of this assessment type in medical education. INTERVENTION: Three US medical schools collaborated to write and score CR-SAQs based on a single vignette. Study participants included faculty question writers (N = 5) and three groups of scorers: faculty content experts (N = 7), faculty non-content experts (N = 6), and fourth-year medical students (N = 7). Structured interviews were performed with question writers and an online survey was administered to scorers to gather information about their process for creating and scoring CR-SAQs. A content analysis was performed on the qualitative data using Bowen's model of feasibility as a framework. To examine inter-rater reliability between the content expert and other scorers, a random selection of fifty student responses from each site were scored by each site's faculty content experts, faculty non-content experts, and student scorers. A holistic rubric (6-point Likert scale) was used by two schools and an analytic rubric (3-4 point checklist) was used by one school. Cohen's weighted kappa (κw) was used to evaluate inter-rater reliability. CONTEXT: This research study was implemented at three US medical schools that are nationally dispersed and have been administering CR-SAQ summative exams as part of their programs of assessment for at least five years. The study exam question was included in an end-of-course summative exam during the first year of medical school. IMPACT: Five question writers (100%) participated in the interviews and twelve scorers (60% response rate) completed the survey. Qualitative comments revealed three aspects of feasibility: practicality (time, institutional culture, teamwork), implementation (steps in the question writing and scoring process), and adaptation (feedback, rubric adjustment, continuous quality improvement). The scorers' described their experience in terms of the need for outside resources, concern about lack of expertise, and value gained through scoring. Inter-rater reliability between the faculty content expert and student scorers was fair/moderate (κw=.34-.53, holistic rubrics) or substantial (κw=.67-.76, analytic rubric), but much lower between faculty content and non-content experts (κw=.18-.29, holistic rubrics; κw=.59-.66, analytic rubric). LESSONS LEARNED: Our findings show that from the faculty perspective it is feasible to include CR-SAQs in summative exams and we provide practical information for medical educators creating and scoring CR-SAQs. We also learned that CR-SAQs can be reliably scored by faculty without content expertise or senior medical students using an analytic rubric, or by senior medical students using a holistic rubric, which provides options to alleviate the faculty burden associated with grading CR-SAQs.

Twelve tips for assessing medical knowledge with open-ended questions: Designing constructed response examinations in medical education.

Medical knowledge examinations employing open-ended (constructed response) items can be useful to assess medical students' factual and conceptual understanding. Modern day curricula that emphasize active learning in small groups and other interactive formats lend themselves to an assessment format that prompts students to share conceptual understanding, explain, and elaborate. The open-ended question examination format can provide faculty with insights into learners' abilities to apply information to clinical or scientific problems, and reveal learners' misunderstandings about essential content. To implement formative or summative assessments with open-ended questions in a rigorous manner, educators must design systems for exam creation and scoring. This includes systems for constructing exam blueprints, items and scoring rubrics, and procedures for scoring and standard setting. Information gained through review of students' responses can guide future educational sessions and curricular changes in a cycle of continuous improvement.

Interprofessional Communication Patterns During Patient Discharges: A Social Network Analysis.

BACKGROUND: Optimal care delivery requires timely, efficient, and accurate communication among numerous providers and their patients, especially during hospital discharge. Little is known about communication patterns during this process. OBJECTIVE: Our aim was to assess the frequency and patterns of communication between patients and providers during patient discharges from a hospital-based medicine unit. DESIGN AND APPROACH: On the day of the patient's discharge, the patient and all healthcare providers involved in the discharge were interviewed using structured questions related to information exchange during the discharge process. Each interview identified the frequency and method of communication between participants, including synchronous (e.g., face-to-face) and asynchronous (e.g., through electronic medical record) routes. Communication patterns were visually diagramed using social network analysis. PARTICIPANTS: Forty-six patients were screened for inclusion in the network analysis. Of those, seven patients who were fully oriented and able to complete an interview and all providers who participated in their care during the discharge were selected for inclusion in the analysis. In all, 72 healthcare professionals contributing to the discharge process were interviewed, including physicians, nurses, therapists, pharmacists, care coordinators, social workers, and nutritionists. KEY RESULTS: Patients' mean age was 63, length-of-stay was 7.8 days, and most (86 %) were discharged to home. On average, 11 roles were involved with each discharge. The majority of communication was synchronous (562 events vs. 469 asynchronous events, p = 0.004). Most communication events occurred between the primary nurse and patient and the care coordinator and primary nurse (mean 3.9 and 2.3 events/discharge, respectively). Participants identified intern physicians as most important in the discharge process, followed by primary nurses and care coordinators. CONCLUSIONS: In patients being discharged from the medicine service, communication was more frequently synchronous, and occurred between intern physicians, primary nurses, and patients. Potential improvements in coordinating patients' discharges are possible by reorganizing systems to optimize efficient communication.

Sleepiness in elderly veterans.

PURPOSE: This observational study seeks to describe the distribution of sleepiness among elderly male veterans and to explain the relationship between sleepiness and age, function, mobility, and depression in this population. METHODS: Veterans who were age 60 or older and had two or more functional limitations based on their activities of daily living or instrumental activities of daily living were recruited in outpatient clinics. They were recruited as part of a longitudinal study and completed questionnaires reported here at the 18-month data collection time point. RESULTS: Veterans report higher sleepiness than normal controls. Self-reported sleepiness scores are similar to those of patients who have been diagnosed with sleep disturbances. CONCLUSIONS: Interventions to identify and support aging veterans with perceived sleepiness should be developed and carefully evaluated.

The internal medicine clerkship and ambulatory learning experiences: results of the 2010 clerkship directors in internal medicine survey.

BACKGROUND: Education in the ambulatory setting should be an integral part of undergraduate medical education. However, previous studies have shown education in this setting has been lacking in medical school. Ambulatory education occurs on some internal medicine clerkships. The extent of this education is unclear. PURPOSE: The purpose of this survey was to assess the structure, curriculum, assessment methods, and barriers to implementation of ambulatory education on the internal medicine clerkship. METHODS: An annual survey of institutional members of the Clerkship Directors in Internal Medicine (CDIM) was done in April 2010. The data were anonymous and descriptive statistics were used to summarize responses. Free text results were analyzed using qualitative techniques. RESULTS: The response rate was 75%. The majority of respondents had a required ambulatory component to the clerkship. Ambulatory experiences distinct from the inpatient internal medicine experience were common (46%). Integration with either the inpatient experiences or other departmental clerkships also occurred. The majority of ambulatory educational experiences were with generalists (74%) and/or subspecialists (45%). The most common assessment tool was the National Board of Medical Examiners (NBME) ambulatory shelf exam. Thematic analysis of the question about how practice based learning was taught elicited four major themes: Not taught; taught in the context of learning evidence based medicine; taught while learning chronic disease management with quality improvement; taught while learning about health care finance. Barriers to implementation included lack of faculty and financial resources. CONCLUSIONS: There have been significant increases in the amount of time dedicated to ambulatory internal medicine. The numbers of medical schools with ambulatory internal medicine education has increased. Integration of the ambulatory experiences with other clerkships such as family medicine occurs. Curriculum was varied but difficulties with dissemination and assessment in these disparate settings was noted. Overall, the results of this study demonstrate increased implementation and recognition of the importance of ambulatory education in internal medicine.

Grade inflation in the internal medicine clerkship: a national survey.

BACKGROUND: Grade inflation is a growing concern, but the degree to which it continues to exist in 3rd-year internal medicine (IM) clerkships is unknown. PURPOSE: The authors sought to determine the degree to which grade inflation is perceived to exist in IM clerkships in North American medical schools. METHODS: A national survey of all Clerkship Directors in Internal Medicine members was administered in 2009. The authors assessed key aspects of grading. RESULTS: Response rate was 64%. Fifty-five percent of respondents agreed that grade inflation exists in the Internal Medicine clerkship at their school. Seventy-eight percent reported it as a serious/somewhat serious problem, and 38% noted students have passed the IM clerkship at their school who should have failed. CONCLUSIONS: A majority of clerkship directors report that grade inflation still exists. In addition, many note students who passed despite the clerkship director believing they should have failed. Interventions should be developed to address both of these problems.

The required written history and physical is alive, but not entirely well, in internal medicine clerkships.

BACKGROUND: Effective written communication is a core competency for medical students, but it is unclear whether or how this skill is evaluated in clinical clerkships. PURPOSE: This study identifies current requirements and practices regarding required written work during internal medicine clerkships. METHODS: In 2010, Clerkship Directors of Internal Medicine (CDIM) surveyed its institutional members; one section asked questions about students' written work. RESULTS were compared to similar, unpublished CDIM 2001 survey questions. RESULTS: Requirements for student-written work were nearly universal (96% in 2001 and 100% in 2010). Only 23% used structured evaluation forms and 16% reported written work was weighted as a percentage of the final grade, although 72% of respondents reported that written work was "factored" into global ratings. CONCLUSIONS: Despite near universal requirements for student written work, structured evaluation was not commonly performed, raising concern about the validity of factoring these assessments into grades.

Reflective writing in the internal medicine clerkship: a national survey of clerkship directors in internal medicine.

BACKGROUND AND PURPOSE: Reflective writing programs have been implemented at many medical schools, but it is unclear to what extent and how they are structured. METHODS: We surveyed the 107 Clerkship Directors of Internal Medicine member institutions on use of reflective writing assignments during the internal medicine clerkship. RESULTS: Eighty-six of 107 (80%) institutional members completed the survey. Thirty-five percent reported having a reflective writing assignment, 48% did not, and 6% did not but were considering starting one within the next 2 years. Of the 30 assignments, most were partially structured (60%), involved small-group discussion (57%), and provided individual student feedback (73%). A minority (30%) contributed to the students' grade. Respondents believed assignments contributed to students' learning in multiple domains, most often Professionalism (97%) and Communication (77%). CONCLUSIONS: Although reflective writing programs were common, variability existed in their structure. Further research is needed to determine how best to implement them.

Simulation in the internal medicine clerkship: a national survey of internal medicine clerkship directors.

BACKGROUND: Simulation in medical education offers the promise of safely and effectively preparing trainees for a variety of tasks encountered in clinical medicine. PURPOSE: The objective was to determine internal medicine (IM) clerkship directors' perceptions of the use of simulations during the medicine clerkship. METHODS: A cross-sectional survey of 110 Clerkship Directors in Internal Medicine (CDIM) institutional members was presented. The survey contained 24 questions including multiple-choice responses with Likert scale and open-ended questions. Descriptive and correlational statistical analyses were performed. A thematic qualitative analysis was performed. RESULTS: Response rate was 78% (86/110). Simulation was used in the IM clerkship in 84% (72) of responding schools. Simulations were used to teach a variety of clinical skills and concepts such as history taking and diagnostic reasoning. The intended goal of simulations was teaching in 39% (28), formative assessment in 49% (35), summative assessment in 38% (27), and compliance with Liaison Council for Medical Education requirements in 19% (14). Most of clerkships (54%, 49) use standardized patients, with 37.5 % (27) using them for formative purposes, whereas 33% (24) used them for summative purposes. Support for simulation was provided by the school of medicine in 57% (41) of clerkships, by the department of medicine in 30% (22) of clerkships. CONCLUSIONS: Simulations are used to teach clinical skills and for formative and summative assessments. Standardized patients and simulation mannequins are the most commonly used simulations types. Further research is needed to assess clerkship directors' needs, knowledge, and use of simulation in specific learning and training contexts.

Analysis and publication rates of Clerkship Directors in Internal Medicine (CDIM) annual meeting abstracts 1995-2005.

BACKGROUND: Abstracts presented at meetings may be a reflection of the meeting's quality. SUMMARY: The goal is to determine purpose, content areas, research design, and subsequent publication rates of abstracts presented at Clerkship Directors in Internal Medicine's annual meetings. Abstracts presented in 1995-2005 were analyzed. A total of 201 abstracts were analyzed and coded. The purpose of the majority of the studies was description (155, 77%), 44 (22%) were justification studies, and 2 (1%) were clarification studies. In all, 109 (54%) assessed the relationship between teaching and student performance. Seven (4%) were studies about how students learn, study, solve problems, obtain medical knowledge, and think critically. Nineteen (10%) were studies about students' noncognitive skills, professionalism, interpersonal skills, and well-being studies. In all, 29 (14%) were measurement studies assessing reliability or validity of assessments. Twenty (10%) focused on career or faculty development pertaining to either faculty or residents. Research design of most studies was descriptive or pre-experimental (175, 87%), some were quasi-experimental (13, 7%), and two thirds (135, 66%) were not published as full articles in peer-reviewed journals. and there was an upward trend in publication rates. CONCLUSIONS: These findings may serve as an indicator of the quality of the educational meeting. It may further stimulate efforts to develop programs to help program attendees achieve scholarly publications for work presented and serve as a benchmark against which future meetings may be judged.

A Novel USMLE Step 1 Projection Model Using a Single Comprehensive Basic Science Self-Assessment Taken During a Brief Intense Study Period.

BACKGROUND: Comprehensive Basic Science Self-Assessments (CBSSAs) offered by the National Board of Medical Examiners (NBME) are used by students to gauge preparedness for the United States Medical Licensing (USMLE) Step 1. Because residency programs value Step 1 scores, students expend many resources attempting to score highly on this exam. We sought to generate a predicted Step 1 score from a single CBSSA taken several days out from a planned exam date to inform student testing and study plans. METHODS: 2016 and 2017 Step 1 test takers at one US medical school were surveyed. The average daily score improvement from CBSSA to Step 1 during the 2016 study period was calculated and used to generate a predicted Step 1 score as well as mean absolute prediction errors (MAPEs). The predictive model was validated on 2017 data. RESULTS: In total, 43 of 61 respondents totaling 141 CBSSAs in 2016 and 37 of 43 respondents totaling 122 CBSSAs in 2017 were included. The final prediction model was [Predicted Step 1 = 292 - (292 - CBSSA score) * 0.987527 ^ (number of days out)]. In 2016, the average difference between predicted and actual scores was -0.81 (10.2) and the MAPE was 7.8. In 2017, 88 (72.1%) and 118 (96.7%) of true Step 1 scores fell within one and two standard deviations of a student's predicted score. There was a MAPE of 7.7. Practice form used (p = 0.19, 0.07) and how far out from actual Step 1 it was taken (p = 0.82, 0.38) were not significant in either year of study. CONCLUSION: This projection model is reasonable for students to use to gauge their readiness for Step 1 while it remains a scored exam and provides a framework for future predictive model generation as the landscape of standardized testing changes in medical education.

Youth Enjoy Science Program at the Case Comprehensive Cancer Center: Increasing Engagement and Opportunity for Underrepresented Minority Students.

The Youth Enjoy Science (YES) Program at the Case Comprehensive Cancer Center is a National Cancer Institute (NCI) R25-funded training grant, designed to increase the pipeline of underrepresented minority (URM) students entering college and pursuing biomedical research and health care careers in the Cleveland Metropolitan and surrounding school districts. The three components of the program include: Learn to Beat Cancer, engaging middle school students and their families; Research to Beat Cancer, designed for high school students and college undergraduates; and Teach to Beat Cancer, focused on enhancing science, technology, engineering, and mathematics (STEM) teaching capacity among high school teachers. This study focuses on Research to Beat Cancer, which, in 2018 enrolled 36 URM students as paid summer scholars. Students were assigned to a faculty mentor, were taught laboratory safety, responsible conduct of research and the scientific method, and then immersed in full-time laboratory cancer research during an eight-week period. Twice each week, students participated in Lunch and Learn Seminars where faculty members provided combined motivational and scientific guidance lectures. In a capstone poster session at the end of the program, students presented their research to peers, medical and graduate students, family members, faculty, community members and leaders. Students' perceptions of the program were reported using descriptive statistics and qualitative thematic analyses. Twenty-four of the 2018 YES students (67%) and 19 (53%) mentors completed the online post-program survey. Opportunity was a major qualitative theme from student and mentor responses. Future research will investigate the long-term impacts of YES, including college enrollment.

Medical informatics in the internal medicine clerkship: Results of a national survey.

BACKGROUND: Medical Informatics (MI) is increasingly a critical aspect of medical education and patient care. AIMS: This study assessed the status of MI training, perception of needs and barriers for the implementation of MI curricula and utilization of information technology (IT) in patient care and medical education. METHOD: The MI questionnaire was a part of the 2006 Clerkship Directors in Internal Medicine survey of 110 institutional members. Descriptive statistics were calculated using Statistical Package for the Social Sciences (SPSS), version 12 and all p-values are two-tailed. RESULTS: Eighty-three (75%) members responded. Out of this, 52, 32.5 and 12% report that students receive MI training for patient care activities during pre-clinical years, third-year internal medicine clerkship or intersession, respectively. House staff critiques (46.4%), patient billing (44.1%), radiographic imaging (40.8%), accessing clinical data (37.3%), and student evaluations (36.1%) were areas in which 35% of respondents use IT 'all the time.' Fifty-one percent of respondents rate the adequacy of training in MI as average. Cost, time and lack of trained faculty were primary barriers for the implementation. CONCLUSIONS: Significant variations exist in timing of MI curricula. IT is utilized more frequently for non-patient activities. Studies are needed to examine the needs, processes and outcomes of MI curricula.

Changes in clinical skills education resulting from the introduction of the USMLE step 2 clinical skills (CS) examination.

BACKGROUND: Step 2 Clinical Skills (CS) was recently introduced into the United States Medical Licensing Examination (USMLE) to ensure that successful candidates for licensure possess the clinical skills to provide safe and effective patient care. AIMS: To explore if medical schools had changed the objectives, content, or emphasis in their pre-clinical curriculum in response to its implementation. METHODS: In April 2005, the Clerkship Directors in Internal Medicine (CDIM) sent an electronic survey to a single member from each medical school with a CDIM member. The survey instrument included 26 pre-clinical curricular questions with nine questions specifically addressing changes in response to implementation of the Step 2 CS. RESULTS: Forty-five percent of respondents reported changes to the Introduction to Clinical Medicine (ICM) course objectives while 39% and 40% reported changes in content and emphasis. Seventy-four percent felt their students were adequately prepared for the Step 2 CS and 18% were unsure. CONCLUSIONS: Over a third of medical schools are implementing changes to the objectives, content, and/or emphasis of their curriculum, at least partially in response to the institution of Step 2 CS.

Developing a mentorship program for clinical researchers.

INTRODUCTION: In academic health centers, the number of physician investigators and the number of research studies headed by clinicians has been declining. The U.S. Institute of Medicine and the National Institutes of Health suggest improved mentoring is important to reversing these trends. METHODS: This is a case study review of the role of mentors in developing and sustaining clinical investigators at the Cleveland Clinic. RESULTS: Issues influencing mentoring relationships at the Cleveland Clinic include whether mentoring contracts are formally or informally related and are agreed on; whether there are scheduled meetings for mentors and protégés; whether there are clearly articulated benchmarks and manageable steps; and whether there is a code of mentorship. CONCLUSIONS: Interactive group formats can lead to development of a code of mentorship and increased awareness of faculty regarding clinical investigation.

Educating for the 21st-Century Health Care System: An Interdependent Framework of Basic, Clinical, and Systems Sciences.

In the face of a fragmented and poorly performing health care delivery system, medical education in the United States is poised for disruption. Despite broad-based recommendations to better align physician training with societal needs, adaptive change has been slow. Traditionally, medical education has focused on the basic and clinical sciences, largely removed from the newer systems sciences such as population health, policy, financing, health care delivery, and teamwork. In this article, authors examine the current state of medical education with respect to systems sciences and propose a new framework for educating physicians in adapting to and practicing in systems-based environments. Specifically, the authors propose an educational shift from a two-pillar framework to a three-pillar framework where basic, clinical, and systems sciences are interdependent. In this new three-pillar framework, students not only learn the interconnectivity in the basic, clinical, and systems sciences but also uncover relevance and meaning in their education through authentic, value-added, and patient-centered roles as navigators within the health care system. Authors describe the Systems Navigation Curriculum, currently implemented for all students at the Penn State College of Medicine, as an example of this three-pillar educational model. Simple adjustments, such as including occasional systems topics in medical curriculum, will not foster graduates prepared to practice in the 21st-century health care system. Adequate preparation requires an explicit focus on the systems sciences as a vital and equal component of physician education.