A Multi-institutional Study of the Feasibility and Reliability of the Implementation of Constructed Response Exam Questions.

PROBLEM: Some medical schools have incorporated constructed response short answer questions (CR-SAQs) into their assessment toolkits. Although CR-SAQs carry benefits for medical students and educators, the faculty perception that the amount of time required to create and score CR-SAQs is not feasible and concerns about reliable scoring may impede the use of this assessment type in medical education. INTERVENTION: Three US medical schools collaborated to write and score CR-SAQs based on a single vignette. Study participants included faculty question writers (N = 5) and three groups of scorers: faculty content experts (N = 7), faculty non-content experts (N = 6), and fourth-year medical students (N = 7). Structured interviews were performed with question writers and an online survey was administered to scorers to gather information about their process for creating and scoring CR-SAQs. A content analysis was performed on the qualitative data using Bowen's model of feasibility as a framework. To examine inter-rater reliability between the content expert and other scorers, a random selection of fifty student responses from each site were scored by each site's faculty content experts, faculty non-content experts, and student scorers. A holistic rubric (6-point Likert scale) was used by two schools and an analytic rubric (3-4 point checklist) was used by one school. Cohen's weighted kappa (κw) was used to evaluate inter-rater reliability. CONTEXT: This research study was implemented at three US medical schools that are nationally dispersed and have been administering CR-SAQ summative exams as part of their programs of assessment for at least five years. The study exam question was included in an end-of-course summative exam during the first year of medical school. IMPACT: Five question writers (100%) participated in the interviews and twelve scorers (60% response rate) completed the survey. Qualitative comments revealed three aspects of feasibility: practicality (time, institutional culture, teamwork), implementation (steps in the question writing and scoring process), and adaptation (feedback, rubric adjustment, continuous quality improvement). The scorers' described their experience in terms of the need for outside resources, concern about lack of expertise, and value gained through scoring. Inter-rater reliability between the faculty content expert and student scorers was fair/moderate (κw=.34-.53, holistic rubrics) or substantial (κw=.67-.76, analytic rubric), but much lower between faculty content and non-content experts (κw=.18-.29, holistic rubrics; κw=.59-.66, analytic rubric). LESSONS LEARNED: Our findings show that from the faculty perspective it is feasible to include CR-SAQs in summative exams and we provide practical information for medical educators creating and scoring CR-SAQs. We also learned that CR-SAQs can be reliably scored by faculty without content expertise or senior medical students using an analytic rubric, or by senior medical students using a holistic rubric, which provides options to alleviate the faculty burden associated with grading CR-SAQs.

A Novel USMLE Step 1 Projection Model Using a Single Comprehensive Basic Science Self-Assessment Taken During a Brief Intense Study Period.

BACKGROUND: Comprehensive Basic Science Self-Assessments (CBSSAs) offered by the National Board of Medical Examiners (NBME) are used by students to gauge preparedness for the United States Medical Licensing (USMLE) Step 1. Because residency programs value Step 1 scores, students expend many resources attempting to score highly on this exam. We sought to generate a predicted Step 1 score from a single CBSSA taken several days out from a planned exam date to inform student testing and study plans. METHODS: 2016 and 2017 Step 1 test takers at one US medical school were surveyed. The average daily score improvement from CBSSA to Step 1 during the 2016 study period was calculated and used to generate a predicted Step 1 score as well as mean absolute prediction errors (MAPEs). The predictive model was validated on 2017 data. RESULTS: In total, 43 of 61 respondents totaling 141 CBSSAs in 2016 and 37 of 43 respondents totaling 122 CBSSAs in 2017 were included. The final prediction model was [Predicted Step 1 = 292 - (292 - CBSSA score) * 0.987527 ^ (number of days out)]. In 2016, the average difference between predicted and actual scores was -0.81 (10.2) and the MAPE was 7.8. In 2017, 88 (72.1%) and 118 (96.7%) of true Step 1 scores fell within one and two standard deviations of a student's predicted score. There was a MAPE of 7.7. Practice form used (p = 0.19, 0.07) and how far out from actual Step 1 it was taken (p = 0.82, 0.38) were not significant in either year of study. CONCLUSION: This projection model is reasonable for students to use to gauge their readiness for Step 1 while it remains a scored exam and provides a framework for future predictive model generation as the landscape of standardized testing changes in medical education.

Youth Enjoy Science Program at the Case Comprehensive Cancer Center: Increasing Engagement and Opportunity for Underrepresented Minority Students.

The Youth Enjoy Science (YES) Program at the Case Comprehensive Cancer Center is a National Cancer Institute (NCI) R25-funded training grant, designed to increase the pipeline of underrepresented minority (URM) students entering college and pursuing biomedical research and health care careers in the Cleveland Metropolitan and surrounding school districts. The three components of the program include: Learn to Beat Cancer, engaging middle school students and their families; Research to Beat Cancer, designed for high school students and college undergraduates; and Teach to Beat Cancer, focused on enhancing science, technology, engineering, and mathematics (STEM) teaching capacity among high school teachers. This study focuses on Research to Beat Cancer, which, in 2018 enrolled 36 URM students as paid summer scholars. Students were assigned to a faculty mentor, were taught laboratory safety, responsible conduct of research and the scientific method, and then immersed in full-time laboratory cancer research during an eight-week period. Twice each week, students participated in Lunch and Learn Seminars where faculty members provided combined motivational and scientific guidance lectures. In a capstone poster session at the end of the program, students presented their research to peers, medical and graduate students, family members, faculty, community members and leaders. Students' perceptions of the program were reported using descriptive statistics and qualitative thematic analyses. Twenty-four of the 2018 YES students (67%) and 19 (53%) mentors completed the online post-program survey. Opportunity was a major qualitative theme from student and mentor responses. Future research will investigate the long-term impacts of YES, including college enrollment.

Twelve tips for assessing medical knowledge with open-ended questions: Designing constructed response examinations in medical education.

Medical knowledge examinations employing open-ended (constructed response) items can be useful to assess medical students' factual and conceptual understanding. Modern day curricula that emphasize active learning in small groups and other interactive formats lend themselves to an assessment format that prompts students to share conceptual understanding, explain, and elaborate. The open-ended question examination format can provide faculty with insights into learners' abilities to apply information to clinical or scientific problems, and reveal learners' misunderstandings about essential content. To implement formative or summative assessments with open-ended questions in a rigorous manner, educators must design systems for exam creation and scoring. This includes systems for constructing exam blueprints, items and scoring rubrics, and procedures for scoring and standard setting. Information gained through review of students' responses can guide future educational sessions and curricular changes in a cycle of continuous improvement.

Educating for the 21st-Century Health Care System: An Interdependent Framework of Basic, Clinical, and Systems Sciences.

In the face of a fragmented and poorly performing health care delivery system, medical education in the United States is poised for disruption. Despite broad-based recommendations to better align physician training with societal needs, adaptive change has been slow. Traditionally, medical education has focused on the basic and clinical sciences, largely removed from the newer systems sciences such as population health, policy, financing, health care delivery, and teamwork. In this article, authors examine the current state of medical education with respect to systems sciences and propose a new framework for educating physicians in adapting to and practicing in systems-based environments. Specifically, the authors propose an educational shift from a two-pillar framework to a three-pillar framework where basic, clinical, and systems sciences are interdependent. In this new three-pillar framework, students not only learn the interconnectivity in the basic, clinical, and systems sciences but also uncover relevance and meaning in their education through authentic, value-added, and patient-centered roles as navigators within the health care system. Authors describe the Systems Navigation Curriculum, currently implemented for all students at the Penn State College of Medicine, as an example of this three-pillar educational model. Simple adjustments, such as including occasional systems topics in medical curriculum, will not foster graduates prepared to practice in the 21st-century health care system. Adequate preparation requires an explicit focus on the systems sciences as a vital and equal component of physician education.

Interprofessional Communication Patterns During Patient Discharges: A Social Network Analysis.

BACKGROUND: Optimal care delivery requires timely, efficient, and accurate communication among numerous providers and their patients, especially during hospital discharge. Little is known about communication patterns during this process. OBJECTIVE: Our aim was to assess the frequency and patterns of communication between patients and providers during patient discharges from a hospital-based medicine unit. DESIGN AND APPROACH: On the day of the patient's discharge, the patient and all healthcare providers involved in the discharge were interviewed using structured questions related to information exchange during the discharge process. Each interview identified the frequency and method of communication between participants, including synchronous (e.g., face-to-face) and asynchronous (e.g., through electronic medical record) routes. Communication patterns were visually diagramed using social network analysis. PARTICIPANTS: Forty-six patients were screened for inclusion in the network analysis. Of those, seven patients who were fully oriented and able to complete an interview and all providers who participated in their care during the discharge were selected for inclusion in the analysis. In all, 72 healthcare professionals contributing to the discharge process were interviewed, including physicians, nurses, therapists, pharmacists, care coordinators, social workers, and nutritionists. KEY RESULTS: Patients' mean age was 63, length-of-stay was 7.8 days, and most (86 %) were discharged to home. On average, 11 roles were involved with each discharge. The majority of communication was synchronous (562 events vs. 469 asynchronous events, p = 0.004). Most communication events occurred between the primary nurse and patient and the care coordinator and primary nurse (mean 3.9 and 2.3 events/discharge, respectively). Participants identified intern physicians as most important in the discharge process, followed by primary nurses and care coordinators. CONCLUSIONS: In patients being discharged from the medicine service, communication was more frequently synchronous, and occurred between intern physicians, primary nurses, and patients. Potential improvements in coordinating patients' discharges are possible by reorganizing systems to optimize efficient communication.

Milestones of critical thinking: a developmental model for medicine and nursing.

Critical thinking is essential to a health professional's competence to assess, diagnose, and care for patients. Defined as the ability to apply higher-order cognitive skills (conceptualization, analysis, evaluation) and the disposition to be deliberate about thinking (being open-minded or intellectually honest) that lead to action that is logical and appropriate, critical thinking represents a "meta-competency" that transcends other knowledge, skills, abilities, and behaviors required in health care professions. Despite its importance, the developmental stages of critical thinking have not been delineated for nurses and physicians. As part of a task force of educators who considered different developmental stage theories, the authors have iteratively refined and proposed milestones in critical thinking. The attributes associated with unreflective, beginning, practicing, advanced, accomplished, and challenged critical thinkers are conceived as independent of an individual's level of training. Depending on circumstances and environmental factors, even the most experienced clinician may demonstrate attributes associated with a challenged thinker. The authors use the illustrative case of a patient with abdominal pain to demonstrate how critical thinking may manifest in learners at different stages of development, analyzing how the learner at each stage applies information obtained in the patient interaction to arrive at a differential diagnosis and plan for evaluation. The authors share important considerations and provide this work as a foundation for the development of effective approaches to teaching and promoting critical thinking and to establishing expectations for learners in this essential meta-competency.

Sleepiness in elderly veterans.

PURPOSE: This observational study seeks to describe the distribution of sleepiness among elderly male veterans and to explain the relationship between sleepiness and age, function, mobility, and depression in this population. METHODS: Veterans who were age 60 or older and had two or more functional limitations based on their activities of daily living or instrumental activities of daily living were recruited in outpatient clinics. They were recruited as part of a longitudinal study and completed questionnaires reported here at the 18-month data collection time point. RESULTS: Veterans report higher sleepiness than normal controls. Self-reported sleepiness scores are similar to those of patients who have been diagnosed with sleep disturbances. CONCLUSIONS: Interventions to identify and support aging veterans with perceived sleepiness should be developed and carefully evaluated.

The internal medicine clerkship and ambulatory learning experiences: results of the 2010 clerkship directors in internal medicine survey.

BACKGROUND: Education in the ambulatory setting should be an integral part of undergraduate medical education. However, previous studies have shown education in this setting has been lacking in medical school. Ambulatory education occurs on some internal medicine clerkships. The extent of this education is unclear. PURPOSE: The purpose of this survey was to assess the structure, curriculum, assessment methods, and barriers to implementation of ambulatory education on the internal medicine clerkship. METHODS: An annual survey of institutional members of the Clerkship Directors in Internal Medicine (CDIM) was done in April 2010. The data were anonymous and descriptive statistics were used to summarize responses. Free text results were analyzed using qualitative techniques. RESULTS: The response rate was 75%. The majority of respondents had a required ambulatory component to the clerkship. Ambulatory experiences distinct from the inpatient internal medicine experience were common (46%). Integration with either the inpatient experiences or other departmental clerkships also occurred. The majority of ambulatory educational experiences were with generalists (74%) and/or subspecialists (45%). The most common assessment tool was the National Board of Medical Examiners (NBME) ambulatory shelf exam. Thematic analysis of the question about how practice based learning was taught elicited four major themes: Not taught; taught in the context of learning evidence based medicine; taught while learning chronic disease management with quality improvement; taught while learning about health care finance. Barriers to implementation included lack of faculty and financial resources. CONCLUSIONS: There have been significant increases in the amount of time dedicated to ambulatory internal medicine. The numbers of medical schools with ambulatory internal medicine education has increased. Integration of the ambulatory experiences with other clerkships such as family medicine occurs. Curriculum was varied but difficulties with dissemination and assessment in these disparate settings was noted. Overall, the results of this study demonstrate increased implementation and recognition of the importance of ambulatory education in internal medicine.

The required written history and physical is alive, but not entirely well, in internal medicine clerkships.

BACKGROUND: Effective written communication is a core competency for medical students, but it is unclear whether or how this skill is evaluated in clinical clerkships. PURPOSE: This study identifies current requirements and practices regarding required written work during internal medicine clerkships. METHODS: In 2010, Clerkship Directors of Internal Medicine (CDIM) surveyed its institutional members; one section asked questions about students' written work. RESULTS were compared to similar, unpublished CDIM 2001 survey questions. RESULTS: Requirements for student-written work were nearly universal (96% in 2001 and 100% in 2010). Only 23% used structured evaluation forms and 16% reported written work was weighted as a percentage of the final grade, although 72% of respondents reported that written work was "factored" into global ratings. CONCLUSIONS: Despite near universal requirements for student written work, structured evaluation was not commonly performed, raising concern about the validity of factoring these assessments into grades.

Grade inflation in the internal medicine clerkship: a national survey.

BACKGROUND: Grade inflation is a growing concern, but the degree to which it continues to exist in 3rd-year internal medicine (IM) clerkships is unknown. PURPOSE: The authors sought to determine the degree to which grade inflation is perceived to exist in IM clerkships in North American medical schools. METHODS: A national survey of all Clerkship Directors in Internal Medicine members was administered in 2009. The authors assessed key aspects of grading. RESULTS: Response rate was 64%. Fifty-five percent of respondents agreed that grade inflation exists in the Internal Medicine clerkship at their school. Seventy-eight percent reported it as a serious/somewhat serious problem, and 38% noted students have passed the IM clerkship at their school who should have failed. CONCLUSIONS: A majority of clerkship directors report that grade inflation still exists. In addition, many note students who passed despite the clerkship director believing they should have failed. Interventions should be developed to address both of these problems.

Hospital medicine in the internal medicine clerkship: results from a national survey.

BACKGROUND: Hospital medicine is growing rapidly. This changing inpatient work force has had consequences on medical education, with an increasing hospitalist presence in resident and student training. Initially met with apprehension, there is growing literature to suggest that hospitalists are perceived to be more effective clinical teachers than non-hospitalists. However, the extent to which hospitalists are involved in teaching Internal Medicine (IM) to medical students is not known. METHODS: In order to determine the role of hospitalists in medical student education within the United States and Canada, we queried clerkship directors in Internal Medicine as part of the 2010 annual Clerkship Directors in Internal Medicine (CDIM) survey. In June 2010, CDIM surveyed its North American institutional members, which represent 110 of 143 Departments of Medicine in the US and Canada. RESULTS: Eight-two of 107 departments responded to the survey (77%). Seventy-five (91%) indicated that hospitalists served as teaching attendings at their teaching hospital. In twenty-two (27%) IM departments, 75% to 100% of students rotate with a hospitalist during their IM clerkships. Thirty-three (42%) departments report that students are directly supervised by in-house hospitalists during their nighttime call requirements. Sixty-six (81%) indicated that academic hospitalists hold educational administrative positions. Hospitalists are significantly less likely to have additional clinical commitment as compared to non-hospitalist teaching attendings (16% vs 53%, (χ 21df = 33.1; P < 0.0001). CONCLUSIONS: Hospitalists are involved in medical student education in the large majority of Departments of Internal Medicine throughout the US and Canada, reflecting the growth of hospital medicine nationally.

Student uncertainties drive teaching during case presentations: more so with SNAPPS.

PURPOSE: To compare the nature of uncertainties expressed by medical students using the six-step SNAPPS technique for case presentations (Summarize history and findings; N>arrow the differential; Analyze the differential; Probe preceptors about uncertainties; Plan management; Select case-related issues for self-study) versus those expressed by students doing customary presentations and to elucidate how preceptors respond. METHOD: The authors performed a secondary analysis in 2009 of data from a 2004-2005 randomized study, comparing SNAPPS users' case presentations with other students' presentations. Authors coded transcriptions of audiotaped presentations to family medicine preceptors for type of student uncertainties, nature of preceptor responses, alignment of preceptor responses with uncertainty types, and expansion of preceptors' responses beyond addressing uncertainties. RESULTS: The analysis included 19 SNAPPS and 41 comparison presentations. SNAPPS students expressed uncertainties in all case presentations, nearly twice as many as the comparison group (χ1df = 12.89, P = .0001). Most SNAPPS users' uncertainties (24/44 [55%]) focused on diagnostic reasoning compared with 9/38 (24%) for comparison students' (χ1df = 8.08, P = .004). Uncertainties about clinical findings and medications/management did not differ significantly between groups. Preceptors responded with teaching aligned with the uncertainties and expanded 24/66 (36%) of their comments. CONCLUSION: Students can drive the content of the teaching they receive based on uncertainties they express to preceptors during case presentations. Preceptors are ready to teach at "the drop of a question" and align their teaching with the content of students' questions; these learning moments-in context and just-in-time-can be created by students.

How and why internal medicine clerkship directors use locally developed, faculty-written examinations: results of a national survey.

PURPOSE: To describe how and why internal medicine clerkship directors (CDs) use locally developed, faculty-written (LFW) examinations. METHOD: In 2009, the Clerkship Directors in Internal Medicine conducted an annual, online, confidential survey of its 107 U.S. and Canadian institutional members, including questions about LFW examinations. Data were analyzed using descriptive statistics and coding of free text. RESULTS: Sixty-nine of 107 members (64.5%) responded. The National Board of Medical Examiners (NBME) examination was administered by 93% (63/68), LFW examinations were used by 33% (22/67), and both types were used by 22% (17/67)-compared with 85%, 36%, and 12% in 2005. LFW examinations were frequently created by the CD alone (9/22; 41%) and consisted of one test (12/22; 52.2%), but some schools gave two (6/22; 26.1%), three (2/22; 8.6%), or four or more (3/22; 13%). Multiple-choice examinations were most common (26/38; 68.4%), followed by short-answer (8/38; 21.1%) and essay (4/38; 10.5%). Most were graded using preestablished criteria; half required a minimum passing score (60% most common). LFW exams were most commonly 5% to 10% of the total grade. Only a minority of CDs reported having reliability estimates or a control group for their exams. Most (70%) reported using LFW exams to cover content felt to be underrepresented by the NBME. CONCLUSIONS: Findings strongly suggest that a minority of internal medicine CDs use LFW examinations, mostly to measure achievement not assessed by the NBME. However, validity evidence is not consistently being gathered, which may limit judgments based on exam results.