How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study.

BACKGROUND: Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace-based assessments (WBAs). Single-institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi-institutional generalisability study to determine the influence of specific factors on reliability of WBAs. METHODS: The authors analysed WBA data obtained for clerkship-level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross-classified random-effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability. RESULTS: Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability. CONCLUSION: The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high-stakes summative entrustment decisions.

Learning by doing and creation of the shared discovery curriculum.

BACKGROUND: The Michigan State College of Human Medicine began as an experiment to teach medical students in community-based settings and to create a primary care workforce for the state. Decades later, CHM faced internal and external challenges that spurred creation of a new curriculum - the Share Discovery Curriculum - founded on learning by doing and other learning theories. METHODS: A curricular design group (CDG) developed guiding principles for reform. Based on this, pedagogies and structures were selected to achieve this vision and developed into a curricular structure. Components of the first-year curriculum were piloted with a group of students and faculty members. RESULTS: Six guiding principles were endorsed, grounded in learning theories such as Dewey's Learning by Doing. Based upon these, several key features of the new curriculum emerged: learning communities; one-on-one coaches for students; symptom-based presentations for content; simulation, authentic clinical tasks, flipped classrooms, and modified practice-based learning as primary teaching modalities; early, integrated clinical and scientific learning; milestones as course learning objectives; and a multidimensional, competency-based assessment system. DISCUSSION: The process and outcomes described here are intended as an exemplar for schools undertaking curricular change. Early stakeholder engagement, faculty development, sustainable administrative systems, and managing complexity are core to the success of such endeavors.

An Exemplar Milestone Framework for Scaffolding a Competency-Based Medical School Curriculum.

A strong competency and milestone framework is imperative for medical schools adopting competency-based education and assessment. Milestones can be used to align what is taught and what students are asked to demonstrate from matriculation to graduation. We describe the creation and implementation of our milestone framework as an exemplar. Supplementary Information: The online version contains supplementary material available at 10.1007/s40670-022-01558-1.

Content and timing of feedback and reflection: a multi-center qualitative study of experienced bedside teachers.

BACKGROUND: Competency-based medical education increasingly recognizes the importance of observation, feedback, and reflection for trainee development. Although bedside rounds provide opportunities for authentic workplace-based implementation of feedback and team-based reflection strategies, this relationship has not been well described. The authors sought to understand the content and timing of feedback and team-based reflection provided by bedside teachers in the context of patient-centered bedside rounds. METHODS: The authors conducted a thematic analysis qualitative study using transcripts from audio-recorded, semi-structured telephone interviews with internal medicine attending physicians (n= 34) identified as respected bedside teachers from 10 academic US institutions (2010-2011). RESULTS: Half of the respondents (50%) were associate/full professors, with an average of 14 years of academic experience. In the context of bedside encounters, bedside teachers reported providing feedback on history-taking, physical-examination, and case-presentation skills, patient-centered communication, clinical decision-making, leadership, teaching skills, and professionalism. Positive feedback about physical-exam skills or clinical decision-making occurred during encounters, positive or constructive team-based feedback occurred immediately following encounters, and individualized constructive feedback occurred in one-on-one settings following rounding sessions. Compared to less frequent, emotionally-charged events, bedside teachers initiated team-based reflection on commonplace "teachable moments" related to patient characteristics or emotions, trainee actions and emotions, and attending physician role modeling. CONCLUSIONS: Bedside teachers use bedside rounds as a workplace-based method to provide assessment, feedback, and reflection, which are aligned with the goals of competency-based medical education. Embedded in patient-centered activities, clinical teachers should be encouraged to incorporate these content- and timing-related feedback and reflection strategies into their bedside teaching.

Addressing mental health issues in primary care: an initial curriculum for medical residents.

OBJECTIVE: Many express concern that modern medicine fails to provide adequate psychosocial and mental health care. Our educational system has not trained the primary care providers who care for most of these patients. Our objective here is to propose a quantum change: prepare residents and students during all years of training so that they are as effective in treating psychosocial and mental health issues as they are medical problems. METHOD: We operationalize this objective, following Kern, by developing an intensive 3-year curriculum in psychosocial and mental health care for medical residents based on models with a strong evidence-base. RESULTS: We report an intensive curriculum that can guide others with similar training interests and also initiate the conversation about how best to prepare residency graduates to provide effective mental health and psychosocial care. CONCLUSION: Identifying specific curricula informs education policy-makers of the specific requirements they will need to meet if psychosocial and mental health training are to improve. PRACTICE IMPLICATIONS: Training residents in mental health will lead to improved care for this very prevalent primary care population.

Identifying and overcoming the barriers to bedside rounds: a multicenter qualitative study.

PURPOSE: The use of bedside rounds in teaching hospitals has declined, despite recommendations from educational leaders to promote this effective teaching strategy. The authors sought to identify reasons for the decrease in bedside rounds, actual barriers to bedside rounds, methods to overcome trainee apprehensions, and proposed strategies to educate faculty. METHOD: A qualitative inductive thematic analysis using transcripts from audio-recorded, semistructured telephone interviews with a purposive sampling of 34 inpatient attending physicians from 10 academic U.S. institutions who met specific inclusion criteria for "bedside rounds" was performed in 2010. Main outcomes were themes pertaining to barriers, methods to overcome trainee apprehensions, and strategies to educate faculty. Quotations highlighting themes are reported. RESULTS: Half of respondents (50%) were associate or full professors, averaging 14 years in academic medicine. Primary reasons for the perceived decline in bedside rounds were physician- and systems related, although actual barriers encountered related to systems, time, and physician-specific issues. To address resident apprehensions, six themes were identified: build partnerships, create safe learning environments, overcome with experience, make bedside rounds educationally worthwhile, respect trainee time, and highlight positive impact on patient care. Potential strategies for educating faculty were identified, most commonly faculty development initiatives, divisional/departmental culture change, and one-on-one shadowing opportunities. CONCLUSIONS: Bedside teachers encountered primarily systems- and time-related barriers and overcame resident apprehensions by creating a learner-oriented environment. Strategies used by experienced bedside teachers can be used for faculty development aimed at promoting bedside rounds.

Are students ready for meaningful use?

BACKGROUND: The meaningful use (MU) of electronic medical records (EMRs) is being implemented in three stages. Key objectives of stage one include electronic analysis of data entered into structured fields, using decision-support tools (e.g., checking drug-drug interactions [DDI]) and electronic information exchange. OBJECTIVE: The authors assessed the performance of medical students on 10 stage-one MU tasks and measured the correlation between students' MU performance and subsequent end-of-clerkship professionalism assessments and their grades on an end-of-year objective structured clinical examination. PARTICIPANTS: Two-hundred and twenty-two third-year medical students on the internal medicine (IM) clerkship. DESIGN/MAIN MEASURES: From July 2010 to February 2012, all students viewed 15 online tutorials covering MU competencies. The authors measured student MU documentation and performance in the chart of a virtual patient using a fully functional training EMR. Specific MU measurements included, adding: a new problem, a new medication, an advanced directive, smoking status, the results of screening tests; and performing a DDI (in which a major interaction was probable), and communicating a plan for this interaction. KEY RESULTS: A total of 130 MU errors were identified. Sixty-eight (30.6%) students had at least one error, and 30 (13.5%) had more than one (range 2-6). Of the 130 errors, 90 (69.2%) were errors in structured data entry. Errors occurred in medication dosing and instructions (18%), DDI identification (12%), documenting smoking status (15%), and colonoscopy results (23%). Students with MU errors demonstrated poorer performance on end-of-clerkship professionalism assessments (r =-0.112, p=0.048) and lower observed structured clinical examination (OSCE) history-taking skills (r =-0.165, p=0.008) and communication scores (r= - 0.173, p=0.006). CONCLUSIONS: MU errors among medical students are common and correlate with subsequent poor performance in multiple educational domains. These results indicate that without assessment and feedback, a substantial minority of students may not be ready to progress to more advanced MU tasks.

Using cloud-based mobile technology for assessment of competencies among medical students.

Valid, direct observation of medical student competency in clinical settings remains challenging and limits the opportunity to promote performance-based student advancement. The rationale for direct observation is to ascertain that students have acquired the core clinical competencies needed to care for patients. Too often student observation results in highly variable evaluations which are skewed by factors other than the student's actual performance. Among the barriers to effective direct observation and assessment include the lack of effective tools and strategies for assuring that transparent standards are used for judging clinical competency in authentic clinical settings. We developed a web-based content management system under the name, Just in Time Medicine (JIT), to address many of these issues. The goals of JIT were fourfold: First, to create a self-service interface allowing faculty with average computing skills to author customizable content and criterion-based assessment tools displayable on internet enabled devices, including mobile devices; second, to create an assessment and feedback tool capable of capturing learner progress related to hundreds of clinical skills; third, to enable easy access and utilization of these tools by faculty for learner assessment in authentic clinical settings as a means of just in time faculty development; fourth, to create a permanent record of the trainees' observed skills useful for both learner and program evaluation. From July 2010 through October 2012, we implemented a JIT enabled clinical evaluation exercise (CEX) among 367 third year internal medicine students. Observers (attending physicians and residents) performed CEX assessments using JIT to guide and document their observations, record their time observing and providing feedback to the students, and their overall satisfaction. Inter-rater reliability and validity were assessed with 17 observers who viewed six videotaped student-patient encounters and by measuring the correlation between student CEX scores and their scores on subsequent standardized-patient OSCE exams. A total of 3567 CEXs were completed by 516 observers. The average number of evaluations per student was 9.7 (±1.8 SD) and the average number of CEXs completed per observer was 6.9 (±15.8 SD). Observers spent less than 10 min on 43-50% of the CEXs and 68.6% on feedback sessions. A majority of observers (92%) reported satisfaction with the CEX. Inter-rater reliability was measured at 0.69 among all observers viewing the videotapes and these ratings adequately discriminated competent from non-competent performance. The measured CEX grades correlated with subsequent student performance on an end-of-year OSCE. We conclude that the use of JIT is feasible in capturing discrete clinical performance data with a high degree of user satisfaction. Our embedded checklists had adequate inter-rater reliability and concurrent and predictive validity.

Mobile technology for the facilitation of direct observation and assessment of student performance.

BACKGROUND: We developed, implemented, and assessed a web-based clinical evaluation application (i.e., CEX app) for Internet-enabled mobile devices, including mobile phones. The app displays problem-specific checklists that correspond to training problems created by the Clerkship Directors in Internal Medicine. PURPOSE: We hypothesized that use of the CEX app for directly observing students' clinical skills would be feasible and acceptable, and would demonstrate adequate reliability and validity. METHODS: Between July 2010 and February 2012, 266 third-year medical students completed 5 to 10 formative CEXs during their internal medicine clerkship. The observers (attendings and residents), who performed the CEX, used the app to guide and document their observations, record their time observing and giving feedback to the students, and their overall satisfaction with the CEX app. Interrater reliability and validity were assessed with 17 observers who viewed 6 videotaped student-patient encounters, and by measuring the correlation between student CEX scores and their scores on subsequent standardized-patient Objective Structured Clinical Examination (OSCE) exams. RESULTS: A total of 2,523 CEXs were completed by 411 observers. The average number of evaluations per student was 9.8 (± 1.8 SD), and the average number of CEXs completed per observer was 6 (± 11.8 SD). Observers spent less than 10 min on 45.3% of the CEXs and 68.6% of the feedback sessions. An overwhelming majority of observers (90.6%) reported satisfaction with the CEX. Interrater reliability was measured at 0.69 among the observers viewing the videotapes, and their ratings discriminated between competent and noncompetent performances. Student CEX grades, however, did not correlate with their end of 3rd-year OSCE scores. CONCLUSIONS: The use of this CEX app is feasible and it captures students' clinical performance data with a high rate of user satisfaction. Our embedded checklists had adequate interrater reliability and concurrent validity. The grades measured on this app, however, were not predictive of subsequent student performance.

The value of bedside rounds: a multicenter qualitative study.

BACKGROUND: Bedside rounds have decreased on teaching services, raising concern about trainees' clinical skills and patient-physician relationships. PURPOSE: We sought to identify recognized bedside teachers' perceived value of bedside rounds to assist in the promotion of bedside rounds on teaching services. METHODS: Authors used a grounded theory, qualitative study design of telephone semistructured interviews with bedside teachers (n = 34) from 10 U.S. institutions (2010-2011). Main outcomes were characteristics of participants, themes pertaining to the perceived value of bedside rounds, and quotations highlighting each respective theme. RESULTS: The mean years in academic medicine was 13.7, and 51% were associate or full professors. Six main themes emerged: (a) skill development for learners (e.g., physical examination, communication, and clinical decision-making skills); (b) observation and feedback; (c) role-modeling; (d) team building among trainees, attending, and patient; (e) improved patient care delivery through combined clinical decision-making and team consensus; and (f) the culture of medicine as patient-centered care, which was embodied in all themes. CONCLUSIONS: Bedside teachers identify potential benefits of bedside rounds, many of which align with national calls to change our approach to medical education. The practice of bedside rounds enables activities essential to high-quality patient care and education.

The art of bedside rounds: a multi-center qualitative study of strategies used by experienced bedside teachers.

BACKGROUND: Medical educators believe bedside rounds are effective for the delivery of patient-centered care, and are necessary in helping trainees acquire competence in clinical care. Although recommendations for bedside rounds have been reported, a recent, systematic assessment of strategies used by current-day bedside teachers was needed to advance knowledge of this teaching method. OBJECTIVE: To identify and understand bedside teachers' 1) preparatory steps, 2) patient selection, and 3) role allocation during the process of bedside rounds. DESIGN: A qualitative inductive thematic analysis using transcripts from audio-recorded, semi-structured telephone interviews. PARTICIPANTS: Internal medicine physicians (n = 34) who perform bedside rounds from ten academic US institutions (2010-2011). APPROACH: A purposive sampling strategy was utilized to identify physicians who were active inpatient attending physicians and met specific inclusion criteria for "bedside rounds." A total of 34 interviews were completed, and each was recorded and transcribed verbatim. A thematic analysis extracted key themes pertaining to the three objectives. KEY RESULTS: Most respondents (51 %) were associate or full professors, with an average of 14 years of academic experience. Attending physicians prepared using trainee-specific, patient-specific and disease-specific information, while also mentally preparing for bedside rounds. They sought trainee buy-in and learning objectives, reviewed expectations and methods to ensure patient comfort, and provided early guidance with bedside encounters. Patients were selected if they required immediate care, were new to the service, or had a high educational value, while patients were deferred if unavailable, unwilling, or unable to communicate. The team members' roles during bedside rounds varied, with trainees being given graduated autonomy with increased experience. CONCLUSIONS: Bedside teachers' methods for preparation, patient selection, and role allocation during bedside rounds enhance trainees' education within the workplace. Strategies used by experienced bedside teachers can be used for faculty development efforts aimed at promoting this activity.

Medicine clerkships and portable computing: a national survey of internal medicine clerkship directors.

BACKGROUND: Portable computers are widely used by medical trainees, but there is a lack of data on how these devices are used in clinical education programs. PURPOSES: The objective is to define the current use of portable computing in internal medicine clerkships and to determine medicine clerkship directors' perceptions of the current value and future importance of portable computing. METHODS: A 2006 national survey of institutional members of the Clerkship Directors in Internal Medicine. RESULTS: Eighty-three of 110 (75%) of institutional members responded. An institutional requirement for portable computing was reported by 32 schools (39%), whereas only 13 (16%) provided students with a portable computer. Between 10 and 31 institutions (12-37%) reported student use for patient care activities (i.e. order entry, writing patient notes) and only 2 to 4 institutions (2-5%) required such use. The majority of respondents (59-95%) reported portable computer use for educational activities (i.e., tracking patient problems, knowledge resource), however, only in 5 to 19 (6-23%) were such educational uses required. Fifty-six respondents (68%) reported that portable computer's "added value" for teaching and 61 (73%) reported that portable computers would be important in meeting clerkship objectives in the next 3 years. Of interest, even among the institutions requiring portable computers, only 50% recommended or required specific software. CONCLUSIONS: Portable computing is required at 39% of allopathic medical schools in the United States. However required portable computing for specific patient care or educational tasks is uncommon. In addition, guidance on specific software exists in only one half of school requiring portable computers, suggesting informal or unstructured uses of required portable computer's in the remaining half. The educational impact of formal institutional requirements for software versus informal "user-defined" applications is unknown.

The implementation of a mobile problem-specific electronic CEX for assessing directly observed student-patient encounters.

BACKGROUND: Facilitating direct observation of medical students' clinical competencies is a pressing need. METHODS: We developed an electronic problem-specific Clinical Evaluation Exercise (eCEX) based on a national curriculum. We assessed its feasibility in monitoring and recording students' competencies and the impact of a grading incentive on the frequency of direct observations in an internal medicine clerkship. Students (n = 56) at three clinical sites used the eCEX and comparison students (n = 56) at three other clinical sites did not. Students in the eCEX group were required to arrange 10 evaluations with faculty preceptors. Students in the second group were required to document a single, faculty observed 'Full History and Physical' encounter with a patient. Students and preceptors were surveyed at the end of each rotation. RESULTS: eCEX increased students' and evaluators' understanding of direct-observation objectives and had a positive impact on the evaluators' ability to provide feedback and assessments. The grading incentive increased the number of times a student reported direct observation by a resident preceptor. CONCLUSIONS: eCEX appears to be an effective means of enhancing student evaluation.

Internal medicine clerkship characteristics associated with enhanced student examination performance.

PURPOSE: To determine which internal medicine (IM) clerkship characteristics are associated with better student examination performance. METHOD: The authors collected data from 17 U.S. medical schools (1,817 students) regarding characteristics of their IM clerkships, including structural characteristics, pedagogical approaches, patient contact, and clinical teacher characteristics. Outcomes of interest were postclerkship National Board of Medical Examiners (NBME) subject examination score, United States Medical Licensing Examination (USMLE) 2 score, and change in score from USMLE 1 to 2. To examine how associations of various clerkship characteristics and examination performance may differ for students of different prior achievement, the authors categorized students into those who scored in the top (1/4) of the cohort on USMLE 1 and the bottom (1/4). The authors conducted analyses at both the school and the individual student levels. RESULTS: In school-level analyses (using a reduced four-variable model), independent variables associated with higher NBME subject examination score were more small-group hours/week and use of community-based preceptors. Greater score increase from USMLE 1 to 2 was associated with students caring for more patients/day. Several variables were associated with enhanced student examination performance at the student level. The most consistent finding was that more patients cared for per day was associated with higher examination performance. More structured learning activities were associated with higher examination scores for students with lower baseline USMLE 1 achievement. CONCLUSION: Certain clerkship characteristics are associated with better student examination performance, the most salient being caring for more patients per day.

Puzzle based teaching versus traditional instruction in electrocardiogram interpretation for medical students--a pilot study.

BACKGROUND: Most medical professionals are expected to possess basic electrocardiogram (EKG) interpretation skills. But, published data suggests that residents' and physicians' EKG interpretation skills are suboptimal. Learning styles differ among medical students; individualization of teaching methods has been shown to be viable and may result in improved learning. Puzzles have been shown to facilitate learning in a relaxed environment. The objective of this study was to assess efficacy of teaching puzzle in EKG interpretation skills among medical students. METHODS: This is a reader blinded crossover trial. Third year medical students from College of Human Medicine, Michigan State University participated in this study. Two groups (n = 9) received two traditional EKG interpretation skills lectures followed by a standardized exam and two extra sessions with the teaching puzzle and a different exam. Two other groups (n = 6) received identical courses and exams with the puzzle session first followed by the traditional teaching. EKG interpretation scores on final test were used as main outcome measure. RESULTS: The average score after only traditional teaching was 4.07 +/- 2.08 while after only the puzzle session was 4.04 +/- 2.36 (p = 0.97). The average improvement after the traditional session was followed up with a puzzle session was 2.53 +/- 1.94 while the average improvement after the puzzle session was followed with the traditional session was 2.08 +/- 1.73 (p = 0.67). The final EKG exam score for this cohort (n = 15) was 84.1 compared to 86.6 (p = 0.22) for a comparable sample of medical students (n = 15) at a different campus. CONCLUSION: Teaching EKG interpretation with puzzles is comparable to traditional teaching and may be particularly useful for certain subgroups of students. Puzzle session are more interactive and relaxing, and warrant further investigations on larger scale.

Using patient encounter logs for mandated clinical encounters in an internal medicine clerkship.

BACKGROUND: Patient encounter logs help assess a student's educational experience. The use of a grading incentive linked to the mandatory documentation of prespecified clinical encounters has been insufficiently studied. PURPOSES: Given this, our objectives were to determine (a) if mandating student exposure to patients with 18 key training problems leads to the successful documentation of these encounters, (b) the degree of difficulty students and clerkship directors experienced in meeting these mandates, (c) the accuracy of mandated log entries, and (d) how often the log entries were questioned or rejected by preceptors. METHODS: Ninety-two 3rd-year internal medicine students and 6 internal medicine clerkship directors at 9 geographically dispersed hospitals and 30 ambulatory sites participated in the study. Over a single academic year, we directly measured the completion rate of mandated logs, the degree of their accuracy as judged against faculty-generated logs of two required videotaped standardized patient encounters, and the percentage of logs that were not validated. We surveyed students and directors on the perceived degree of difficulty in meeting mandated requirements. RESULTS: Ninety-eight percent of students met our mandated requirements and 93.8% of students found it "easy" or "very easy" to meet this requirement. The amount of estimated time spent by clerkship directors helping students meet mandated requirements for the entire year was 4.5 hr. The accuracy of submitted logs was 77%; however, almost all inaccurate log entries were "validated" by preceptors. CONCLUSIONS: Mandating encounters is effective in assuring that students document encounters with patients who present with specific clinical problems. The accuracy of our students' mandated logs is similar to previously published data. However, even inaccurate logs were rarely questioned or rejected by preceptors.

Just in time: technology to disseminate curriculum and manage educational requirements with mobile technology.

BACKGROUND: Learning objectives intended to guide clinical education may be of limited usefulness if they are unavailable to students when interacting with patients. DESCRIPTION: We developed, implemented, and evaluated a Web-based process to disseminate the Clerkship Directors of Internal Medicine curricular objectives to students via handheld computers and for students to upload patient logs to a central database. EVALUATION: We delivered this program to all students in our geographically dispersed system, with minimal technological problems. The total number of "hits" on curricular objectives was 8,932 (averaging 149 per student or approximately 2.7 times daily). The average number of "hits" per problem was 470, ranging from 18 for smoking cessation to 1,784 for chest pain. The total number of patient problems logged by students was 9,579, and 91% of students met our prespecified criteria for numbers and types of patients. CONCLUSIONS: Dissemination and use of curricular learning objectives and related tools is enhanced with mobile technology.

A national collaboration to disseminate skills for outpatient teaching in internal medicine: program description and preliminary evaluation.

The shift of clinical care and teaching to outpatient settings has challenged ambulatory and community-based teachers. To address this challenge, U.S. internal medicine organizations devised "Faculty Development for General Internal Medicine: Generalist Faculty Teaching in Ambulatory Settings," a national program to train leaders to create local faculty development projects. In 1999, teams from all 386 internal medicine training institutions were invited to apply. Participation required an acceptable plan for a local project and inclusion of an institutional leader, residency or clerkship director, and a community-based faculty member on the project team. Team members attended one of three national training conferences held in 1999 and 2000 that included plenary sessions, workshops, and team meetings. Participants were invited to a wrap-up conference to present their accomplishments. One hundred ten teams from 57 university and 53 non-university hospitals attended the training conferences; 412 (93%) participants returned conference evaluations. All sessions were rated highly. Participants preferred workshops and team meetings to plenary sessions. Two hundred thirty-five (57%) would have recommended the training conference to colleagues as an outstanding experience; 148 (36%) as a good experience; and 25 (6%) as a satisfactory experience. Forty-nine teams (122 participants) returned for the wrap up conference where 35 teams presented their local faculty development projects. Cost per team trained was US$11,818. This program demonstrated a national desire for training in teaching skills, reached a broad audience of ambulatory-based clinical teachers, provided highly rated faculty development conferences in teaching skills, and facilitated development of a variety of local projects at modest expense. Partnerships were forged between academic leaders and community-based teachers.

Sources of measurement error in an ECG examination: implications for performance-based assessments.

OBJECTIVE: To assess the sources of measurement error in an electrocardiogram (ECG) interpretation examination given in a third-year internal medicine clerkship. DESIGN: Three successive generalizability studies were conducted. (1) Multiple faculty rated student responses to a previously administered exam. (2) The rating criteria were revised and study 1 was repeated. (3) The examination was converted into an extended matching format including multiple cases with the same underlying cardiac problem. RESULTS: The discrepancies among raters (main effects and interactions) were dwarfed by the error associated with case specificity. The largest source of the differences among raters was in rating student errors of commission rather than student errors of omission. Revisions in the rating criteria may have helped increase inter-rater reliability slightly however, due to case specificity, it had little impact on the overall reliability of the exam. The third study indicated the majority of the variability in student performance across cases was in performance across cases within the same type of cardiac problem rather than between different types of cardiac problems. CONCLUSIONS: Case specificity was the overwhelming source of measurement error. The variation among cases came mainly from discrepancies in performance between examples of the same cardiac problem rather than from differences in performance across different types of cardiac problems. This suggests it is necessary to include a large number of cases even if the goal is to assess performance on only a few types of cardiac problems.

Outcomes of a national faculty development program in teaching skills: prospective follow-up of 110 medicine faculty development teams.

BACKGROUND: Awareness of the need for ambulatory care teaching skills training for clinician-educators is increasing. A recent Health Resources and Services Administration (HRSA)-funded national initiative trained 110 teams from U.S. teaching hospitals to implement local faculty development (FD) in teaching skills. OBJECTIVE: To assess the rate of successful implementation of local FD initiatives by these teams. METHODS: A prospective observational study followed the 110 teams for up to 24 months. Self-reported implementation, our outcome, was defined as the time from the training conference until the team reported that implementation of their FD project was completely accomplished. Factors associated with success were assessed using Kaplan-Meier analysis. RESULTS: The median follow-up was 18 months. Fifty-nine of the teams (54%) implemented their local FD project and subsequently trained over 1,400 faculty, of whom over 500 were community based. Teams that implemented their FD projects were more likely than those that did not to have the following attributes: met more frequently (P=.001), had less turnover (P=.01), had protected time (P=.01), rated their likelihood of success high (P=.03), had some project or institutional funding for FD (P=.03), and came from institutions with more than 75 department of medicine faculty (P=.03). The cost to the HRSA was $22,033 per successful team and $533 per faculty member trained. CONCLUSIONS: This national initiative was able to disseminate teaching skills training to large numbers of faculty at modest cost. Smaller teaching hospitals may have limited success without additional support or targeted funding.