Validity evidence for the clinical communication skills assessment tool (CCSAT) from 9 years of implementation in a high stakes medical student OSCE.

OBJECTIVES: Communication and other clinical skills are routinely assessed in medical schools using Objective Structured Clinical Examinations (OSCEs) so routinely that it can be difficult to monitor and maintain validity. We report on the accumulation of validity evidence for the Clinical Communication Skills Assessment Tool (CCSAT) based on its use with 9 cohorts of medical students in a high stakes OSCE. METHODS: We describe the implementation of the CCSAT including information on the underlying model, the tool's items, domains, scales and scoring, and its role in curriculum. Internal structure is explored through item, internal consistency, and confirmatory factor analyses. Evidence for CCSAT validity is synthesized within prevailing frameworks (Messick12 and Kane13) based on continuous quality improvement and use of the CCSAT for feedback, remediation, curricular design, and research. RESULTS: Implementation of the CCSAT over time has facilitated our communication skills curriculum and training. Thoughtful case development and investment in standardized patient training has contributed to data quality. Item analysis supports our behaviorally anchored scale (not done, partly and well done) and the skills domains suggested by an a priori evidence-based clinical communication model were confirmed via analysis of actual student data. Evidence synthesized across the frameworks suggests consistent validity of the CCSAT for generalization inferences (that it captures the construct), responsiveness (sensitivity to change/difference), content validity/internal structure, relationships to other variables, and consequences/implications. More evidence is needed to strengthen validity of CCSAT scores for understanding extrapolation inferences and real-world implications. CONCLUSIONS AND PRACTICE IMPLICATIONS: This pragmatic approach to evaluating validity within a program of assessment serves as a model for medical schools seeking to continuously monitor the quality of clinical skill assessments, a need made particularly relevant since the US NBME no longer requires the Step 2 Clinical Skills exam, leaving individual schools with the responsibility for ensuring graduates have acquired the requisite core clinical skills. We document strong evidence for CCSAT validity over time and across cohorts as well as areas for improvement and further examination.

What Do Pediatric Subinterns Say About Their Learning and Assessment? A Qualitative Analysis of Individual Learning Plans.

OBJECTIVE: To perform a qualitative content analysis of learning and assessment strategies that pediatric subinterns describe in Individualized Learning Plans (ILPs) and to explore barriers and facilitators to their learning. METHODS: We analyzed ILPs from medical students enrolled in pediatric subinternships at 10 US medical schools that utilized a standardized curriculum and were recruited to reflect diversity in geographic location, funding, and enrollment. Students used an ILP to record 3 or more selected learning objectives, rationale for selection, and reflection on learning and assessment strategies. Investigators used the constant comparative method to perform a content analysis of the ILPs, grouping codes into themes, and verifying relationships between codes within themes. RESULTS: Two hundred and four ILPs that included student reflections on 850 learning objectives were analyzed. Content was analyzed in 5 categories: rationale for selecting objectives, learning strategies, assessment strategies, challenges to learning, and facilitators of learning. Students showed strong commitment to individualized, self-directed learning, developed a wide range of creative learning strategies, and relied heavily on self-reflection to assess their progress. The learning environment both helped and hindered students' ability to make and assess progress on their selected learning objectives. CONCLUSIONS: Through ILP-guided reflection and a formal curriculum, students can choose well-justified learning objectives and demonstrate resourcefulness and independence in developing self-directed learning and assessment strategies. The strategies that students identified in this study provide a menu of learning and assessment options for subinterns. Identified challenges and facilitators of learning provide guidance for educators who seek to enhance the clinical learning environment.

Faculty Development Offered by US Medical Schools: A National Survey of Pediatric Educators.

INTRODUCTION: There are limited data on the status of faculty development (FD) in the United States. Through a national survey of pediatric educators, we explored the frequency and topics of FD on teaching skills offered at US medical schools, as well as the strategies' schools use to encourage and track participation. METHODS: Five piloted questions were included in the 2017 Annual Council on Medical Student Education in Pediatrics Survey. We used descriptive statistics. RESULTS: Ninety-seven (66%) of the 148 surveyed US medical schools responded to at least one FD question. Ninety-eight percent of respondents reported being offered FD on teaching, with 97% of those respondents reporting that FD occurred at least annually. A variety of FD topics were reported, with feedback and precepting being most common. Incentives included continuing medical education (CME credit) (39%) and being relieved of clinical duties (23%). However, 29% reported little support for FD. Only 20% of schools reported their department tracked FD participation outside the department. DISCUSSION: Our data suggest that the majority of medical schools offer FD on teaching skills at least yearly, with a variety of topics. Institutions utilize a variety of incentives for participation. However, a significant minority of respondents reported little support for FD. Further, departments rarely track faculty FD participation.

Does a measure of Medical Professional Identity Formation predict communication skills performance?

OBJECTIVE: To validate an approach to measuring professional identity formation (PIF), we explore if the Professional Identity Essay (PIE), a stage score measure of medical professional identity (PI), predicts clinical communication skills. METHODS: Students completed the PIE during medical school orientation and a 3-case Objective Structured Clinical Exam (OSCE) where standardized patients reliably assessed communication skills in 5 domains. Using mediation analyses, relationships between PIE stage scores and communication skills were explored. RESULTS: For the 351 (89%) consenting students, controlling for individual characteristics, there were increases in patient counseling (6.5%, p<0.01), information gathering (4.3%, p = 0.01), organization and management (4.1%, p = 0.02), patient assessment (3.6%, p = 0.04), and relationship development (3.5%, p = 0.03) skills for every half stage increase in PIE score. The communication skills of lower socio-economic status (SES) students are indirectly impacted by their slightly higher PIE stage scores. CONCLUSION: Higher PIE stage scores are associated with higher communication skills and lower SES. PRACTICE IMPLICATIONS: PIE predicts critical clinical skills and identifies how SES and other characteristics indirectly impact future clinical performance, providing validity evidence for using PIE as a tool in longitudinal formative academic coaching, program and curriculum evaluation, and research.

Collecting Validity Evidence: A Hands-on Workshop for Medical Education Assessment Instruments.

Introduction: There is an increasing call for developing validity evidence in medical education assessment. The literature lacks a practical resource regarding an actual development process. Our workshop teaches how to apply principles of validity evidence to existing assessment instruments and how to develop new instruments that will yield valid data. Methods: The literature, consensus findings of curricula and content experts, and principles of adult learning guided the content and methodology of the workshop. The workshop underwent stringent peer review prior to presentation at one international and three national academic conferences. In the interactive workshop, selected domains of validity evidence were taught with sequential cycles of didactics, demonstration, and deliberate practice with facilitated feedback. An exercise guide steered participants through a stepwise approach. Using Likert-scale items and open-response questions, an evaluation form rated the workshop's effectiveness, captured details of how learners reached the objectives, and determined participants' plans for future work. Results: The workshop demonstrated generalizability with successful implementation in diverse settings. Sixty-five learners, the majority being clinician-educators, completed evaluations. Learners rated the workshop favorably for each prompt. Qualitative comments corroborated the workshop's effectiveness. The active application and facilitated feedback components allowed learners to reflect in real time as to how they were meeting a particular objective. Discussion: This feasible and practical educational intervention fills a literature gap by showing the medical educator how to apply validity evidence to both existing and in-development assessment instruments. Thus, it holds the potential to significantly impact learner and, subsequently, patient outcomes.

Conducting Quantitative Medical Education Research: From Design to Dissemination.

Rigorous medical education research is critical to effectively develop and evaluate the training we provide our learners. Yet many clinical medical educators lack the training and skills needed to conduct high-quality medical education research. We offer guidance on conducting sound quantitative medical education research. Our aim is to equip readers with the key skills and strategies necessary to conduct successful research projects, highlighting new concepts and controversies in the field. We utilize Glassick's criteria for scholarship as a framework to discuss strategies to ensure that the research question of interest is worthy of further study and how to use existing literature and conceptual frameworks to strengthen a research study. Through discussions of the strengths and limitations of commonly used study designs, we expose the reader to particular nuances of these decisions in medical education research and discuss outcomes generally focused on, as well as strategies for determining the significance of consequent findings. We conclude with information on critiquing research findings and preparing results for dissemination to a broad audience. Practical planning worksheets and comprehensive tables illustrating key concepts are provided in order to guide researchers through each step of the process. Medical education research provides wonderful opportunities to improve how we teach our learners, to satisfy our own intellectual curiosity, and ultimately to enhance the care provided to patients.

Evaluation of a National Pediatric Subinternship Curriculum Implemented Through Individual Learning Plans.

OBJECTIVE: The Council on Medical Student Education in Pediatrics and Association of Pediatric Program Directors developed a Pediatric Subinternship (CAPS) curriculum for use with an individualized learning plan (ILP). The authors determined which learning objectives (LOs) pediatric subinterns selected when provided the CAPS curriculum, summarized students' self-reported progress, and determined feasibility of ILPs in subinternship. METHODS: Students from 10 medical schools completed a standardized ILP during pediatric subinternship. Students listed ≥3 LOs using CAPS curriculum as a guide and self-assessed their progress. Students reviewed ILPs with faculty preceptors; preceptors completed questionnaires on time and effort spent. Authors mapped student LOs to CAPS curriculum objectives and grouped in Accreditation Council for Graduate Medical Education competency domains. RESULTS: Two hundred four students documented 850 LOs. Authors mapped student LOs to 61 of the 69 CAPS objectives (88%). Students most commonly chose Patient Care LOs, with the top 3 related to oral presentations, time management, and management plans. Student LOs not in CAPS addressed nutrition, child development, test interpretation, and cost. No students chose LOs related to health disparities, shared decision making, informed consent, or patient safety. Students self-reported significant progress on most LOs (73%). Faculty met with students ≥1 time and 93% met for a total of ≤1 hour. According to faculty, students required little or no help completing ILPs. CONCLUSIONS: Students chose a wide range of LOs when provided the CAPS curriculum. Revision to include additional student-identified LOs would enhance CAPS curriculum's comprehensiveness. Using this curriculum with an ILP during subinternship is feasible, but gaps between educator-identified and student-identified objectives require further exploration.

Professional Identity Formation in medical school: One measure reflects changes during pre-clerkship training.

This article was migrated. The article was marked as recommended. Professional Identity Formation (PIF), the process of internalizing a profession's core values and beliefs, is an explicit goal of medical education. The Professional Identity Essay (PIE), a developmental measure of the extent to which individuals have a complex and self-defined understanding of their professional role, is a tool to both study and scaffold PIF. PIE staging has internal reliability and response process validity and correlates with a validated measure of moral reasoning. In this study, we investigate whether PIF, as measured by PIE, changes during pre-clerkship training. Medical students in the class of 2019 completed the PIE during orientation to medical school (PIE#1) and 15 months later, during orientation to clerkships (PIE#2), to the same prompts. These written responses are PIF-staged by an expert rater. On average, PIF scores reveal that 46% of the group remained at the same stage as they were on entry to medical school, 42% scored at a higher stage of PIF, and 15% of students scored at a lower stage of PIF after pre-clerkship training. This result suggests that medical students are heterogeneous with respect to the development of their medical PIF early in medical school training.

Constructing a Shared Mental Model for Faculty Development for the Core Entrustable Professional Activities for Entering Residency.

In 2014, the Association of American Medical Colleges identified 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs), which are activities that entering residents might be expected to perform without direct supervision. This work included the creation of an interinstitutional concept group focused on faculty development efforts, as the processes and tools for teaching and assessing entrustability in undergraduate medical education (UME) are still evolving. In this article, the authors describe a conceptual framework for entrustment that they developed to better prepare all educators involved in entrustment decision making in UME. This framework applies to faculty with limited or longitudinal contact with medical students and to those who contribute to entrustment development or render summative entrustment decisions.The authors describe a shared mental model for entrustment that they developed, based on a critical synthesis of the EPA literature, to serve as a guide for UME faculty development efforts. This model includes four dimensions for Core EPA faculty development: (1) observation skills in authentic settings (workplace-based assessments), (2) coaching and feedback skills, (3) self-assessment and reflection skills, and (4) peer guidance skills developed through a community of practice. These dimensions form a conceptual foundation for meaningful faculty participation in entrustment decision making.The authors also differentiate between the UME learning environment and the graduate medical education learning environment to highlight distinct challenges and opportunities for faculty development in UME settings. They conclude with recommendations and research questions for future Core EPA faculty development efforts.

Qualitative abstracts at the Pediatric Academic Societies meeting: are they less likely to be accepted for presentation?

OBJECTIVE: To determine the proportion of abstracts submitted to the 2010 Pediatric Academic Societies (PAS) meeting that were exclusively qualitative, and to examine whether these abstracts were more or less likely than all others to be designated as platform, poster, or publish only. METHODS: The database of abstracts submitted to the 2010 PAS meeting was searched using qualitative terms. Authors reviewed abstracts to identify exclusively qualitative abstracts. The proportions and mean score ± standard deviation for qualitative abstracts and those designated platform, poster, or publish only were calculated. Student's t test was used to analyze mean differences; pairwise comparisons and odds ratios (ORs) were used to examine differences in the disposition of qualitative versus all other abstracts. The main outcome was the proportion of abstracts designated as platform, poster, or publish only. RESULTS: Of 4057 abstracts, 1.6% used only qualitative methods. Scores for qualitative and all other abstracts were 4.26 ± 0.60 and 3.88 ± 0.92, respectively (P < .001). The disposition of qualitative abstracts among platform, poster, and publish only differed from all other abstracts (P = .04). Compared with all others, qualitative abstracts had lower odds of platform presentation (OR 0.36; 95% confidence interval [CI] 0.15-0.91), demonstrated a nonsignificant trend toward publish only (OR 1.6; 95% CI 0.95-2.7), and were equally likely to be posters (OR 1.1; 95% CI 0.65-1.8). CONCLUSIONS: Compared with all other abstracts, qualitative abstracts were about 3 times less likely to be chosen for platform presentations; in addition, they demonstrated a trend toward greater odds of publish only. These findings may be the result of inferior quality or an inadequate review process.

Assessment of medical student clinical reasoning by "lay" vs physician raters: inter-rater reliability using a scoring guide in a multidisciplinary objective structured clinical examination.

BACKGROUND: To determine whether a "lay" rater could assess clinical reasoning, interrater reliability was measured between physician and lay raters of patient notes written by medical students as part of an 8-station objective structured clinical examination. METHODS: Seventy-five notes were rated on core elements of clinical reasoning by physician and lay raters independently, using a scoring guide developed by physician consensus. Twenty-five notes were rerated by a 2nd physician rater as an expert control. Kappa statistics and simple percentage agreement were calculated in 3 areas: evidence for and against each diagnosis and diagnostic workup. RESULTS: Agreement between physician and lay raters for the top diagnosis was as follows: supporting evidence, 89% (κ = .72); evidence against, 89% (κ = .81); and diagnostic workup, 79% (κ = .58). Physician rater agreement was 83% (κ = .59), 92% (κ = .87), and 96% (κ = .87), respectively. CONCLUSIONS: Using a comprehensive scoring guide, interrater reliability for physician and lay raters was comparable with reliability between 2 expert physician raters.

Teaching patient communication skills to medical students: a review of randomized controlled trials.

Tools to examine the effects of teaching interventions across a variety of studies are needed. The authors perform a meta-analysis of 24 randomized controlled trials evaluating the effects of teaching on medical students' patient communication skills. Study quality is rated using a modified Jadad score, and standardized mean difference effect size (d) measures are calculated. Fifteen of 24 studies have sufficient data for analysis. Students' ability to establish rapport improves after teaching. The effects are large when the teaching intervention was small group discussion (n = 5) or giving structured feedback on a student-patient interview (n = 6). A similar effect of teaching is seen on student data gathering skills (n = 5). Teaching medical students patient communication skills using small group discussion or providing feedback on a student-patient interview results in improvement in student performance.