E-learning in graduate medical education: survey of residency program directors.

BACKGROUND: E-learning-the use of Internet technologies to enhance knowledge and performance-has become a widely accepted instructional approach. Little is known about the current use of e-learning in postgraduate medical education. To determine utilization of e-learning by United States internal medicine residency programs, program director (PD) perceptions of e-learning, and associations between e-learning use and residency program characteristics. METHODS: We conducted a national survey in collaboration with the Association of Program Directors in Internal Medicine of all United States internal medicine residency programs. RESULTS: Of the 368 PDs, 214 (58.2%) completed the e-learning survey. Use of synchronous e-learning at least sometimes, somewhat often, or very often was reported by 85 (39.7%); 153 programs (71.5%) use asynchronous e-learning at least sometimes, somewhat often, or very often. Most programs (168; 79%) do not have a budget to integrate e-learning. Mean (SD) scores for the PD perceptions of e-learning ranged from 3.01 (0.94) to 3.86 (0.72) on a 5-point scale. The odds of synchronous e-learning use were higher in programs with a budget for its implementation (odds ratio, 3.0 [95% CI, 1.04-8.7]; P = .04). CONCLUSIONS: Residency programs could be better resourced to integrate e-learning technologies. Asynchronous e-learning was used more than synchronous, which may be to accommodate busy resident schedules and duty-hour restrictions. PD perceptions of e-learning are relatively moderate and future research should determine whether PD reluctance to adopt e-learning is based on unawareness of the evidence, perceptions that e-learning is expensive, or judgments about value versus effectiveness.

Rating the Quality of Entrustable Professional Activities: Content Validation and Associations with the Clinical Context.

BACKGROUND: Entrustable professional activities (EPAs) have been developed to assess resident physicians with respect to Accreditation Council for Graduate Medical Education (ACGME) competencies and milestones. Although the feasibility of using EPAs has been reported, we are unaware of previous validation studies on EPAs and potential associations between EPA quality scores and characteristics of educational programs. OBJECTIVES: Our aim was to validate an instrument for assessing the quality of EPAs for assessment of internal medicine residents, and to examine associations between EPA quality scores and features of rotations. DESIGN: This was a prospective content validation study to design an instrument to measure the quality of EPAs that were written for assessing internal medicine residents. PARTICIPANTS: Residency leadership at Mayo Clinic, Rochester participated in this study. This included the Program Director, Associate program directors and individual rotation directors. INTERVENTIONS: The authors reviewed salient literature. Items were developed to reflect domains of EPAs useful for assessment. The instrument underwent further testing and refinement. Each participating rotation director created EPAs that they felt would be meaningful to assess learner performance in their area. These 229 EPAs were then assessed with the QUEPA instrument to rate the quality of each EPA. MAIN MEASURES: Performance characteristics of the QUEPA are reported. Quality ratings of EPAs were compared to the primary ACGME competency, inpatient versus outpatient setting and specialty type. KEY RESULTS: QUEPA tool scores demonstrated excellent reliability (ICC range 0.72 to 0.94). Higher ratings were given to inpatient versus outpatient (3.88, 3.66; p = 0.03) focused EPAs. Medical knowledge EPAs scored significantly lower than EPAs assessing other competencies (3.34, 4.00; p < 0.0001). CONCLUSIONS: The QUEPA tool is supported by good validity evidence and may help in rating the quality of EPAs developed by individual programs. Programs should take care when writing EPAs for the outpatient setting or to assess medical knowledge, as these tended to be rated lower.

Hospitalist career decisions among internal medicine residents.

BACKGROUND: Hospital medicine is a rapidly growing field of internal medicine. However, little is known about internal medicine residents' decisions to pursue careers in hospital medicine (HM). OBJECTIVE: To identify which internal medicine residents choose a career in HM, and describe changes in this career choice over the course of their residency education. DESIGN: Observational cohort using data collected from the annual Internal Medicine In-Training Examination (IM-ITE) survey. PARTICIPANTS: 16,781 postgraduate year 3 (PGY-3) North American internal medicine residents who completed the annual IM-ITE survey in 2009-2011, 9,501 of whom completed the survey in all 3 years of residency. MAIN MEASURES: Self-reported career plans for individual residents during their postgraduate year 1 (PGY-1), postgraduate year 2 (PGY-2) and PGY-3. KEY RESULTS: Of the 16,781 graduating PGY-3 residents, 1,552 (9.3 %) reported HM as their ultimate career choice. Of the 951 PGY-3 residents planning a HM career among the 9,501 residents responding in all 3 years, 128 (13.5 %) originally made this decision in PGY-1, 192 (20.2 %) in PGY-2, and 631 (66.4 %) in PGY-3. Only 87 (9.1 %) of these 951 residents maintained a career decision of HM during all three years of residency education. CONCLUSIONS: Hospital medicine is a reported career choice for an important proportion of graduating internal medicine residents. However, the majority of residents do not finalize this decision until their final year.

General medicine vs subspecialty career plans among internal medicine residents.

CONTEXT: Current medical training models in the United States are unlikely to produce sufficient numbers of general internists and primary care physicians. Differences in general internal medicine (GIM) career plans between internal medicine residency program types and across resident demographics are not well understood. OBJECTIVE: To evaluate the general medicine career plans of internal medicine residents and how career plans evolve during training. DESIGN, SETTING, AND PARTICIPANTS: A study of US internal medicine residents using an annual survey linked to the Internal Medicine In-Training Examination taken in October of 2009-2011 to evaluate career plans by training program, sex, and medical school location. Of 67,207 US eligible categorical and primary care internal medicine residents, 57,087 (84.9%) completed and returned the survey. Demographic data provided by the National Board of Medical Examiners were available for 52,035 (77.4%) of these residents, of whom 51,390 (76.5%) responded to all survey items and an additional 645 (1.0%) responded to at least 1 survey item. Data were analyzed from the 16,781 third-year residents (32.2%) in this sample. MAIN OUTCOME MEASURES: Self-reported ultimate career plans of internal medicine residents. RESULTS: A GIM career plan was reported by 3605 graduating residents (21.5%). A total of 562 primary care program (39.6%) and 3043 categorical (19.9%) residents reported GIM as their ultimate career plan (adjusted odds ratio [AOR], 2.76; 99% CI, 2.35-3.23; P < .001). Conversely, 10 008 categorical (65.3%) and 745 primary care program (52.5%) residents reported a subspecialty career plan (AOR, 1.90; 99% CI, 1.62-2.23; P < .001). GIM career plans were reported more frequently by women than men (26.7% vs 17.3%, respectively; AOR, 1.69; 99% CI, 1.53-1.87; P < .001). US medical graduates were slightly more likely to report GIM career plans than international medical graduates (22.0% vs 21.1%, respectively; AOR, 1.76; 99% CI, 1.50-2.06; P < .001). Within primary care programs, US medical graduates were much more likely to report GIM career plans than international medical graduates (57.3% vs 27.3%, respectively; AOR, 3.48; 99% CI, 2.58-4.70; P < .001). Compared with their counterparts, maintaining a first-year GIM career plan over the course of their training was more likely among primary care program residents (68.2% vs 52.3%; AOR, 1.81; 99% CI, 1.25-2.64; P < .001), women (62.4% vs 47.2%; AOR, 1.75; 99% CI, 1.34-2.29; P < .001), and US medical graduates (60.9% vs 49.2%; AOR, 1.48; 99% CI, 1.13-1.93; P < .001). CONCLUSION: Reported GIM career plans were markedly less common than subspecialty career plans among internal medicine residents, including those in primary care training programs, and differed according to resident sex, medical school location, and program type.

What do we mean by web-based learning? A systematic review of the variability of interventions.

OBJECTIVES: Educators often speak of web-based learning (WBL) as a single entity or a cluster of similar activities with homogeneous effects. Yet a recent systematic review demonstrated large heterogeneity among results from individual studies. Our purpose is to describe the variation in configurations, instructional methods and presentation formats in WBL. METHODS: We systematically searched MEDLINE, EMBASE, ERIC, CINAHL and other databases (last search November 2008) for studies comparing a WBL intervention with no intervention or another educational activity. From eligible studies we abstracted information on course participants, topic, configuration and instructional methods. We summarised this information and then purposively selected and described several WBL interventions that illustrate specific technologies and design features. RESULTS: We identified 266 eligible studies. Nearly all courses (89%) used written text and most (55%) used multimedia. A total of 32% used online communication via e-mail, threaded discussion, chat or videoconferencing, and 9% implemented synchronous components. Overall, 24% blended web-based and non-computer-based instruction. Most web-based courses (77%) employed specific instructional methods, other than text alone, to enhance the learning process. The most common instructional methods (each used in nearly 50% of courses) were patient cases, self-assessment questions and feedback. We describe several studies to illustrate the range of instructional designs. CONCLUSIONS: Educators and researchers cannot treat WBL as a single entity. Many different configurations and instructional methods are available for WBL instructors. Researchers should study when to use specific WBL designs and how to use them effectively.

Assessment of Training in Health Disparities in US Internal Medicine Residency Programs.

Importance: Health disparities continue to exist despite the call to increase education of health care practitioners. An assessment of health disparities education has not been previously studied in a national cohort. Objective: To describe and compare the curriculum on health disparities from the perspective of program directors and perceptions of training among internal medicine residents. Design, Setting, and Participants: This cross-sectional survey study used a survey of US internal medicine program directors, the 2015 Association of Program Directors in Internal Medicine annual survey, which included questions about health disparities curriculum, and a 1-time survey of US internal medicine residents that asked questions related to their training in health disparities on the American College of Physicians 2015 Internal Medicine In-Training Examination. All internal medicine program directors who were members of Association of Program Directors in Internal Medicine (368 of 396 accredited programs), and internal medicine residents who took the Internal Medicine In-Training Examination were eligible. Final analysis of the merged data set was completed in 2018. Exposures: Questions were included on the annual Association of Program Directors in Internal Medicine survey and the Internal Medicine In-Training Examination. Main Outcomes and Measures: Program directors reported on presence of health disparities curriculum, educational methods, quality, barriers and challenges to curriculum, and Clinical Learning Environment Review. Residents reported whether they received training and quality of the training in health disparities. Results: A total of 227 program directors (response rate, 61.7%) and 22 723 residents (response rate, 87.2%) responded to the surveys. A total of 90 program directors (39.6%) reported a curriculum in health disparities, but among these, only 16 program directors (17.8%) felt quality of their education was very good or excellent. In more than half of the programs (52 programs [55.9%]), outcomes of the curriculum were not measured. After merging, the combined data set included 18 883 residents from 366 APDIM member programs with 225 program director responses. Among these, 13 251 residents (70.2%) reported some training in caring for patients at risk for health disparities. Of residents who reported receiving training, 10 494 (79.2%) rated the quality as very good or excellent. Conclusions and Relevance: These findings suggest that that despite the Clinical Learning Environment Review mandate and Accreditation Council for Graduate Medical Education required competencies for training in health disparities, relatively few internal medicine programs in the US provided educational curriculum. Additionally, the existence of health disparities curricula in internal medicine training programs was not associated with resident's perception of training or its quality.

Effect of rater training on reliability and accuracy of mini-CEX scores: a randomized, controlled trial.

BACKGROUND: Mini-CEX scores assess resident competence. Rater training might improve mini-CEX score interrater reliability, but evidence is lacking. OBJECTIVE: Evaluate a rater training workshop using interrater reliability and accuracy. DESIGN: Randomized trial (immediate versus delayed workshop) and single-group pre/post study (randomized groups combined). SETTING: Academic medical center. PARTICIPANTS: Fifty-two internal medicine clinic preceptors (31 randomized and 21 additional workshop attendees). INTERVENTION: The workshop included rater error training, performance dimension training, behavioral observation training, and frame of reference training using lecture, video, and facilitated discussion. Delayed group received no intervention until after posttest. MEASUREMENTS: Mini-CEX ratings at baseline (just before workshop for workshop group), and four weeks later using videotaped resident-patient encounters; mini-CEX ratings of live resident-patient encounters one year preceding and one year following the workshop; rater confidence using mini-CEX. RESULTS: Among 31 randomized participants, interrater reliabilities in the delayed group (baseline intraclass correlation coefficient [ICC] 0.43, follow-up 0.53) and workshop group (baseline 0.40, follow-up 0.43) were not significantly different (p = 0.19). Mean ratings were similar at baseline (delayed 4.9 [95% confidence interval 4.6-5.2], workshop 4.8 [4.5-5.1]) and follow-up (delayed 5.4 [5.0-5.7], workshop 5.3 [5.0-5.6]; p = 0.88 for interaction). For the entire cohort, rater confidence (1 = not confident, 6 = very confident) improved from mean (SD) 3.8 (1.4) to 4.4 (1.0), p = 0.018. Interrater reliability for ratings of live encounters (entire cohort) was higher after the workshop (ICC 0.34) than before (ICC 0.18) but the standard error of measurement was similar for both periods. CONCLUSIONS: Rater training did not improve interrater reliability or accuracy of mini-CEX scores. CLINICAL TRIALS REGISTRATION: clinicaltrials.gov identifier NCT00667940

Associations Among Practice Variation, Clinician Characteristics, and Care Algorithm Usage: A Multispecialty Vignette Study.

The objective was to quantitatively evaluate clinician characteristics associated with unwarranted practice variation, and how clinical care algorithms influence this variation. Participants (142 physicians, 53 nurse practitioners, and 9 physician assistants in family medicine, internal medicine, and cardiology) described their management of 4 clinical vignettes, first based on their own practice (unguided), then using care algorithms (guided). The authors quantitatively estimated variation in management. Cardiologists demonstrated 17% lower variation in unguided responses than generalists (fold-change 0.83 [95% confidence interval (CI) 0.68, 0.97]), and those who agreed that practice variation can realistically be reduced had 16% lower variation than those who did not (fold-change 0.84 [CI, 0.71, 0.99]). A 17% reduction in variation was observed for guided responses compared with baseline (unguided) responses (fold-change 0.83 [CI, 0.76, 0.90]). Differences were otherwise similar across clinician subgroups and attitudes. Unwarranted practice variation was similar across most clinician subgroups. The authors conclude that care algorithms can reduce variation in management.

Internet-based learning in the health professions: a meta-analysis.

CONTEXT: The increasing use of Internet-based learning in health professions education may be informed by a timely, comprehensive synthesis of evidence of effectiveness. OBJECTIVES: To summarize the effect of Internet-based instruction for health professions learners compared with no intervention and with non-Internet interventions. DATA SOURCES: Systematic search of MEDLINE, Scopus, CINAHL, EMBASE, ERIC, TimeLit, Web of Science, Dissertation Abstracts, and the University of Toronto Research and Development Resource Base from 1990 through 2007. STUDY SELECTION: Studies in any language quantifying the association of Internet-based instruction and educational outcomes for practicing and student physicians, nurses, pharmacists, dentists, and other health care professionals compared with a no-intervention or non-Internet control group or a preintervention assessment. DATA EXTRACTION: Two reviewers independently evaluated study quality and abstracted information including characteristics of learners, learning setting, and intervention (including level of interactivity, practice exercises, online discussion, and duration). DATA SYNTHESIS: There were 201 eligible studies. Heterogeneity in results across studies was large (I(2) > or = 79%) in all analyses. Effect sizes were pooled using a random effects model. The pooled effect size in comparison to no intervention favored Internet-based interventions and was 1.00 (95% confidence interval [CI], 0.90-1.10; P < .001; n = 126 studies) for knowledge outcomes, 0.85 (95% CI, 0.49-1.20; P < .001; n = 16) for skills, and 0.82 (95% CI, 0.63-1.02; P < .001; n = 32) for learner behaviors and patient effects. Compared with non-Internet formats, the pooled effect sizes (positive numbers favoring Internet) were 0.10 (95% CI, -0.12 to 0.32; P = .37; n = 43) for satisfaction, 0.12 (95% CI, 0.003 to 0.24; P = .045; n = 63) for knowledge, 0.09 (95% CI, -0.26 to 0.44; P = .61; n = 12) for skills, and 0.51 (95% CI, -0.24 to 1.25; P = .18; n = 6) for behaviors or patient effects. No important treatment-subgroup interactions were identified. CONCLUSIONS: Internet-based learning is associated with large positive effects compared with no intervention. In contrast, effects compared with non-Internet instructional methods are heterogeneous and generally small, suggesting effectiveness similar to traditional methods. Future research should directly compare different Internet-based interventions.

Practice variation and practice guidelines: Attitudes of generalist and specialist physicians, nurse practitioners, and physician assistants.

OBJECTIVE: To understand clinicians' beliefs about practice variation and how variation might be reduced. METHODS: We surveyed board-certified physicians (N = 178), nurse practitioners (N = 60), and physician assistants (N = 12) at an academic medical center and two community clinics, representing family medicine, general internal medicine, and cardiology, from February-April 2016. The Internet-based questionnaire ascertained clinicians' beliefs regarding practice variation, clinical practice guidelines, and costs. RESULTS: Respondents agreed that practice variation should be reduced (mean [SD] 4.5 [1.1]; 1 = strongly disagree, 6 = strongly agree), but agreed less strongly (4.1 [1.0]) that it can realistically be reduced. They moderately agreed that variation is justified by situational differences (3.9 [1.2]). They strongly agreed (5.2 [0.8]) that clinicians should help reduce healthcare costs, but agreed less strongly (4.4 [1.1]) that reducing practice variation would reduce costs. Nearly all respondents (234/249 [94%]) currently depend on practice guidelines. Clinicians rated differences in clinician style and experience as most influencing practice variation, and inaccessibility of guidelines as least influential. Time to apply standards, and patient decision aids, were rated most likely to help standardize practice. Nurse practitioners and physicians assistants (vs physicians) and less experienced (vs senior) clinicians rated more favorably several factors that might help to standardize practice. Differences by specialty and academic vs community practice were small. CONCLUSIONS: Clinicians believe that practice variation should be reduced, but are less certain that this can be achieved. Accessibility of guidelines is not a significant barrier to practice standardization, whereas more time to apply standards is viewed as potentially helpful.

Flipped Classrooms in Graduate Medical Education: A National Survey of Residency Program Directors.

PURPOSE: To begin to quantify and understand the use of the flipped classroom (FC)-a progressive, effective, curricular model-in internal medicine (IM) education in relation to residency program and program director (PD) characteristics. METHOD: The authors conducted a survey that included the Flipped Classroom Perception Instrument (FCPI) in 2015 regarding programs' use and PDs' perceptions of the FC model. RESULTS: Among the 368 IM residency programs, PDs at 227 (61.7%) responded to the survey and 206 (56.0%) completed the FCPI. Regarding how often programs used the FC model, 34 of the 206 PDs (16.5%) reported "never"; 44 (21.4%) reported "very rarely"; another 44 (21.4%) reported "somewhat rarely"; 59 (28.6%) reported "sometimes"; 16 (7.8%) reported "somewhat often"; and 9 (4.4%) reported "very often." The mean FCPI score (standard deviation [SD]) for the in-class application factor (4.11 [0.68]) was higher (i.e., more favorable) than for the preclass activity factor (3.94 [0.65]) (P < .001). FC perceptions (mean [SD]) were higher among younger PDs (≤ 50 years, 4.12 [0.62]; > 50 years, 3.94 [0.61]; P = .04) and women compared with men (4.28 [0.56] vs. 3.91 [0.62]; P < .001). PDs with better perceptions of FCs had higher odds of using FCs (odds ratio, 4.768; P < .001). CONCLUSIONS: Most IM programs use the FC model at least to some extent, and PDs prefer the interactive in-class components over the independent preclass activities. PDs who are women and younger perceived the model more favorably.

Clinicians' guide to new tools and features of PubMed.

Practicing clinicians need to have the skills required to obtain up- to-date medical information to address both the expansion of scientific knowledge and patients' increasing use of the Internet. PubMed (www.pubmed.gov) allows clinicians free access to the largest biomedical resource available. This article is the third in a Mayo Clinic Proceedings series designed specifically to help clinicians unlock the tools and information available through this valuable resource.

Web-based learning in residents' continuity clinics: a randomized, controlled trial.

PURPOSE: To determine whether internal medicine residents prefer learning from Web-based (WB) modules or printed material, and to compare the effect of these teaching formats on knowledge. METHOD: The authors conducted a randomized, controlled, crossover study in the internal medicine resident continuity clinics of the Mayo School of Graduate Medical Education during the 2002-03 academic year. Participants studied two topics of ambulatory medicine using WB modules and two topics using paper practice guidelines in randomly assigned sequences. Primary outcomes were format preference (assessed by an end-of-course questionnaire) and score changes from pre- to postintervention tests of knowledge. RESULTS: A total of 109 consented and 75 (69%) completed the postintervention test. Fifty-seven of 73 (78% [95% CI, 67-86%]) preferred the WB format (p < .001). Test scores improved for both formats (67.7% to 75.0% for WB, 66.0% to 73.3% for paper), but score change was not different between formats both before (p = .718) and after (p = .080) adjusting for topic, clinic site, study group, postgraduate year, and gender. Residents spent less time on WB modules (mean = 47 +/- 26 minutes) than paper (mean = 59 +/- 35, p = .024). Difficulties with passwords limited their use of WB modules for 71% (59-80%) of residents. CONCLUSION: No difference was found between WB and paper-based formats in knowledge-test score change, but residents preferred learning with WB modules and spent less time doing so. Passwords appeared to impede use of WB modules. WB learning is effective, well accepted, and efficient. Research should focus on aspects of WB instruction that will enhance its power as a learning tool and better define its role in specific settings.

Searching the medical literature using PubMed: a tutorial.

Staying current with advances in medicine is becoming a major challenge for clinicians. Access to updated repositories of medical information and the expertise to locate relevant information within them are becoming necessary clinical skills. PubMed (http://www.pubmedgov) provides free access to the largest biomedical resource available and is updated daily. Clinicians can use this resource to find answers to focused clinical questions quickly and efficiently. The purpose of this article is to assist clinicians in the development of the basic skills required to use PubMed to make informed clinical decisions.

Effect of publication bias on retrieval bias.

Publication bias is the main etiologic factor in retrieval bias. The authors measured the influence a positive study outcome had on housestaff's selecting the study for presentation.

Patient satisfaction with point-of-care international normalized ratio testing and counseling in a community internal medicine practice.

Point-of-care international normalized ratio (POC INR) testing is increasingly used to monitor anticoagulant therapy. This study assessed patient satisfaction with registered nurse--managed POC INR testing in a primary care internal medicine practice. One hundred eighty-seven of the first 216 (87%) patients using the program responded to the survey. The respondents were surveyed for satisfaction after they had experienced the nurse-managed POC INR system at the Mayo Clinic, Rochester, Minnesota, for one month. Eighty-eight percent of patients indicated that they were very satisfied or satisfied with the POC INR system. The authors conclude that nurse-managed POC INR testing is quick, convenient, less painful, and more satisfying for patients compared with traditional venipuncture and telephone follow-up.

5 ways statistics can fool you--tips for practicing clinicians.

Published literature suggests that many clinicians are not fully equipped to evaluate and apply research reports for the care of their patients. In this article, we introduce and illustrate five basic statistical concepts that can significantly impact the interpretation of the medical literature and its application to the care of patients, drawing examples from the vaccine literature: (i) consider clinical and statistical significance separately, (ii) evaluate absolute risks rather than relative risks, (iii) examine confidence intervals rather than p values, (iv) use caution when considering isolated significant p values in the setting of multiple testing, and (v) keep in mind that statistically nonsignificant results may not exclude clinically important benefits or harms. These tips may help busy clinicians better interpret the increasingly overwhelming amount of medical literature they are faced with in their daily practices.