Utilizing natural language processing to analyze student narrative reflections for medical curriculum improvement.

MOTIVATION: Medical curricula improvement is an ongoing process to keep material relevant and improve the student's learning experience to better prepare them for patient care. Many programs utilize end-of-year evaluations, but these frequently have low response rates and lack actionable feedback. We hypothesized that student reflections written during a fourth year Sub-Internship could be used retrospectively to mine additional information as feedback for future curriculum adjustments. However, reflections contain a large amount of narrative content that would require a cumbersome and essentially infeasible manual review process for busy medical education faculty. METHODS: We developed a Natural Language Processing (NLP) pipeline to automatically identify common themes and topics present in the set of reflective writings that could be used to improve the curriculum. The dataset contains required responses to a faculty issued question submitted between August 2016 and July 2018 about challenges experienced during the medical students fourth year Sub-Internship. RESULTS: Eleven distinct topics were identified, with several being subsequently addressed in future iterations of the curriculum. CONCLUSION: Utilizing NLP on reflective writings was able to identify areas of curriculum improvement, and the NLP results provided a quick and easy way to explore the main themes and challenges expressed by students.

Mapping an undergraduate medical education curriculum against national and international palliative care reference learning objectives.

BACKGROUND: The teaching of palliative care competencies is an essential component of undergraduate medical education. There is significant variance in the palliative care content delivered in undergraduate medical curricula, revealing the utility of reference standards to guide curricular development and assessment. To evaluate our university's undergraduate palliative care teaching, we undertook a curriculum mapping exercise, comparing official learning objectives to the national Educating Future Physicians in Palliative and End-of-Life Care (EFPPEC) and the international Palliative Education Assessment Tool (PEAT) reference objectives. METHODS: Multiple assessors independently compared our university's UGME learning objectives with EFPPEC and PEAT reference objectives to determine the degree-of-coverage. Visual curriculum maps were created to depict in which part of the curriculum each objective is delivered and by which medical specialty. RESULTS: Of 122 EFPPEC objectives, 55 (45.1%) were covered fully, 42 (34.4%) were covered partially, and 25 (20.5%) were not covered by university objectives. Of 89 PEAT objectives, 40 (44.9%) were covered fully, 35 (39.3%) were covered partially, and 14 (15.7%) were not covered by university objectives. CONCLUSIONS: The majority of EFPPEC and PEAT reference objectives are fully or partially covered in our university's undergraduate medical curriculum. Our approach could serve as a guide for others who endeavour to review their universities' specialty-specific medical education against reference objectives. Future curriculum development should target the elimination of identified gaps and evaluate the attainment of palliative care competencies by medical learners.

Comparison of faculty and student perceptions of sexual and gender minority content in a preclerkship medical curriculum.

BACKGROUND: Sexual and gender minority (SGM) persons experience stark health disparities. Efforts to mitigate disparities through medical education have met some success. However, evaluations have largely focused on subjective perspectives rather than objective measures. This study aimed to quantify Boston University School of Medicine's sexual and gender minority (SGM) education through surveys of course directors (CDs) and medical students regarding where SGM topics were taught in the preclerkship medical curriculum. Responses were compared to identify concordance between faculty intention and student perceptions regarding SGM education. METHODS: A cross-sectional survey was distributed to preclerkship CDs and current medical students in Spring 2019 and 2021, respectively, regarding where in the mandatory preclerkship curriculum CDs deliberately taught and where first- and second-year students recalled having learned 10 SGM topic domains. RESULTS: 64.3% of CDs (n = 18), 47.0% of the first-year class (n = 71), and 67.3% of the second-year class (n = 101) responded to the surveys. Results indicate that, as anticipated, deliberate SGM teaching correlates with greater student recall as students recalled topics that were reported by CDs as intentionally taught at a significantly higher rate compared to those not intentionally taught (32.0% vs. 15.3%; p < 0.01). Students most commonly recalled learning SGM-related language and terminology, which is likely partly but not entirely attributed to curricular modifications and faculty development made between distribution of the faculty and student surveys, indicating the importance of all faculty being trained in appropriate SGM terminology and concepts. Discordance between faculty intention and student recall of when topics were taught reveals opportunities to enhance the intentionality and impact of SGM teaching. CONCLUSIONS: Students perceive and recall SGM content that is not listed as learning objectives, and all faculty who utilize this material in their teachings should receive foundational training and be thoughtful about how information is framed. Faculty who intentionally teach SGM topics should be explicit and direct about the conclusions they intend students to draw from their curricular content.

Development of an OIE Harmonized Day 1 Competency-Based Veterinary School Curriculum in Ethiopia: A Partnership Model for Curriculum Evaluation and Implementation.

The University of Gondar College of Veterinary Medicine and Animal Sciences (UoG-CVMASc) and the Ohio State University College of Veterinary Medicine (OSU-CVM) developed an objective methodology to assess the curriculum of veterinary institutions and implement changes to create a curriculum that is harmonized with OIE standards while also covering the needs and realities of Gondar and Ethiopia. The process, developed under the sponsorship of the World Organisation for Animal Health (OIE) Veterinary Education Twinning Programme, is outlined in this article with the hope that it can be applied by other countries wishing to improve national veterinary services (VS) through the improvement of their academic programs. The plan created by the UoG-OSU Twinning team consisted of an in-depth curriculum assessment and development process, which entailed three consecutive stages. Stage 1 (Curriculum Assessment) included the design and development of an Evaluation Tool for OIE Day 1 Graduating Veterinarian Competencies in recent graduates, and the mapping and evaluation of the current UoG-CVMASc curriculum based on the OIE Veterinary Education Core Curriculum. Stage 2 (Curriculum Development) consisted of the identification and prioritization of possible solutions to address identified curriculum gaps as well as the development of an action plan to revise and update the curriculum. Finally, Stage 3 (Curriculum Implementation) focused on the process to launch the new curriculum. In September 2017, 53 first-year students started the professional program at the UoG-CVMASc as the first cohort to be accepted into the newly developed OIE Harmonized Curriculum, the first of its kind in Africa.

A Global Approach to Promoting Evidence-Based Practice Knowledge: Validating the Translated Version of the Evidence-Based Practice Knowledge Assessment in Nursing Into Spanish.

BACKGROUND: Government and professional organizations have issued a call for evidence-based practice (EBP), prompting the development of competencies and expectations for knowledge, skills, and attitudes. However, EBP is still inconsistently implemented. To advance the science of nursing globally, educators and leaders need to use consistent, rigorous evaluation methods to assess the EBP knowledge of students and clinicians. AIMS: This study sought to translate the Evidence-based Practice Knowledge Assessment in Nursing (EKAN) instrument into Spanish, evaluate the psychometric performance of the newly translated instrument, and establish a baseline measure of EBP knowledge of baccalaureate nursing (BSN) students for future program improvement. METHODS: This study utilized a cross-sectional, descriptive, correlational design. A total of 139 BSN students were recruited from a single study site in the Dominican Republic. Subjects completed the 20-item EKAN-Spanish, which was translated and back-translated to promote semantic accuracy. RESULTS AND FINDINGS: Infit and outfit statistics for the sample centered around 1.0 (weighted mean square = 0.978; unweighted mean square = 0.988), indicating strong evidence of validity. The individual items on the EKAN-Spanish showed item reliability of 0.94. The mean EBP knowledge score was 6.52 (SD = 2.03) with scores ranging from 2 to 12 points (20 points maximum). Differential item functioning (DIF) analysis supported translational accuracy. DISCUSSION: This study provides validation support for an objective assessment of EBP knowledge in the Spanish language and will be useful for future studies of EBP in over 21 Spanish-speaking countries. LINKING EVIDENCE TO ACTION: While EBP is a global nursing concept, there has been no instrument to objectively measure EBP knowledge across levels of education in academe and practice in Spanish-speaking countries until just recently. The availability of a validated instrument will promote global consistency in assessing EBP knowledge. Further, this will promote a collaborative approach in teaching EBP to nursing students and practicing nurses.

Prognosis communication with older patients with multimorbidity: Assessment after an educational intervention.

This study aimed to assess how internal medicine residents incorporated prognosis to inform clinical decisions and communicated prognosis in primary care visits with older patients with multimorbidity after an educational intervention, and resident and patient perspectives regarding these visits. Assessment used mixed-methods. The authors assessed the frequency and content of prognosis discussions through residents' self-report and qualitative content analysis of audio-recorded clinic visits. The authors assessed the residents' perceived effect of incorporating prognosis on patient care and patient relationship through a resident survey. The authors assessed the patients' perceived quality of communication and trust in physicians through a patient survey. The study included 21 clinic visits that involved 12 first-year residents and 21 patients. Residents reported incorporating patients' prognoses to inform clinical decisions in 13/21 visits and perceived positive effects on patient care (in 11/13 visits) and patient relationship (in 7/13 visits). Prognosis communication occurred in 9/21 visits by self-report, but only in six of these nine visits by content analysis of audio-recordings. Patient ratings were high regardless of whether or not prognosis was communicated. In summary, after training, residents often incorporated patients' prognoses to inform clinical decisions, but sometimes did so without communicating prognosis to the patients. Residents and patients reported positive perceptions regarding the visits.

The Utility of Multiple-Choice Assessment in Current Medical Education: A Critical Review.

In recent years, healthcare education providers have boasted about a conscious shift towards increasing clinical competence via assessment tests that promote more active learning. Despite this, multiple-choice questions remain amongst the most prevalent forms of assessment. Various literature justifies the use of multiple-choice testing by its high levels of validity and reliability. Education providers also benefit from requiring fewer resources and costs in the development of questions and easier adaptivity of questions to compensate for neurodiversity. However, when testing these (and other) variables via a structured approach in terms of their utility, it is elucidated that these advantages are largely dependent on the quality of the questions that are written, the level of clinical competence that is to be attained by learners and the impact of negating confounding variables such as differential attainment. Attempts at improving the utility of multiple-choice question testing in modern healthcare curricula are discussed in this review, as well as the impact of these modifications on performance.

Experiences of teachers in vocational programmes in special needs schools, City of Cape Town.

Background: Vocational programmes run by teachers in the special needs school context can play a significant role in the vocational development of learners with severe intellectual disability (SID). This study aimed to answer the question 'what are the challenges faced by teachers in the implementation of vocational programmes in selected public special needs schools for learners with SID in the Metropolitan (Metro) District within the City of Cape Town?' Objectives: The objectives were to describe the challenges as perceived by participants, to highlight common and contrasting challenges in the different schools and to share recommendations on support needed. Method: A qualitative descriptive study was conducted. A combination of purposive and snowball sampling strategies was used to select six teachers from six special needs schools. One-on-one semi-structured interviews with teachers were performed. An interview schedule was used as a tool and all interviews were transcribed and translated into English verbatim. Thematic analysis was applied. Results: The findings showed that teachers encounter inadequate resources, a lack of training, and poor support systems. This study highlights the issues of existing policy and the lack of a mandatory policy on vocational programmes in South Africa. Conclusion: The participants' experiences added to the existing literature by providing valuable insights into the obstacles teachers encounter in this relatively new curriculum. A multifaceted policy framework that is well funded and implemented is much needed to address the challenges identified. Contribution: The findings may contribute to the development and strengthening of policies on vocational programmes within the South African context.

A systematic review of general surgery robotic training curriculums.

Background: As of the most recent surveys of resident programs in 2018, only slightly more than half of programs have formal robotic training curriculums implemented. Fewer programs have further assessed their own curriculum and its benefit. Method: We conducted a PubMed/MEDLINE literature search for robotic surgery curriculums and those that had assessment of their programs. Results: A total of 11 studies were reviewed. When reviewed in chronological order, there has been a progression towards more robotic specific objective data analysis as opposed to subjective surveying. There is a wide variation in curriculums, but simulation use is pervasive. Conclusions: Our review makes evident two important concepts-there is great variety in training curriculums and there is great benefit in implementation. The importance is in establishment of what makes resident training effective and supports the adaptable and successful surgeon. This may come from an adaptable curriculum but a structured test-out assessment.

Critical Care Ultrasound Competency of Fellows and Faculty in Pulmonary and Critical Care Medicine: A Nationwide Survey.

Purpose: Competency assessment standards for Critical Care Ultrasonography (CCUS) for Graduate Medical Education (GME) trainees in pulmonary/critical care medicine (PCCM) fellowship programs are lacking. We sought to answer the following research questions: How are PCCM fellows and teaching faculty assessed for CCUS competency? Which CCUS teaching methods are perceived as most effective by program directors (PDs) and fellows. Methods: Cross-sectional, nationwide, electronic survey of PCCM PDs and fellows in accredited GME training programs. Results: PDs and fellows both reported the highest rates of fellow competence to use CCUS for invasive procedural guidance, but lower rates for assessment of deep vein thrombosis and abdominal organs. 54% and 90% of PDs reported never assessing fellows or teaching faculty for CCUS competency, respectively. PDs and fellows perceived hands-on workshops and directly supervised CCUS exams as more effective learning methods than unsupervised CCUS archival with subsequent review and self-directed learning. Conclusions: There is substantial variation in CCUS competency assessment among PCCM fellows and teaching faculty nationwide. The majority of training programs do not formally assess fellows or teaching faculty for CCUS competence. Guidelines are needed to formulate standardized competency assessment tools for PCCM fellowship programs.

Does exposure to research experiences have different learning outcomes than prior exposure to lab techniques in non-research settings?

A large body of literature has established the benefits of undergraduate research experiences via the traditional apprenticeship model. More recently, several studies have shown that many of these benefits can be recapitulated in course-based undergraduate research experiences (CUREs) that are more scalable and easier for students to participate in, compared to the apprenticeship-based research experiences. Many Biology curricula also incorporate more traditional laboratory courses, where students learn to use common laboratory techniques through guided exercises with known outcomes. Indeed, many programs across the nation provide such programs or courses for students early in their careers, with a view toward increasing student interest and engagement in Biology. While there is general consensus that all lab experiences have some benefits for students, very few studies have examined whether either research experiences or learning biological techniques in more traditional lab courses directly impacts student performance in lecture courses. Here, we show that prior familiarity with laboratory techniques does not improve student performance in a lecture course, even if these techniques are directly related to content being taught in the course. However, having prior research experience improves performance in the course, irrespective of whether the research experience included the use of course-related laboratory techniques.

Course-based Undergraduate Research Experiences (CUREs) in General Education Courses.

While much of the promotion for undergraduate research (UR) originates from the natural sciences, this high-impact practice should also occur in social science to prepare students for graduate school/ the workforce and should be integrated into lower-division general education courses. Our study examines content and skills gained by students from two course-based undergraduate research experiences (CUREs) in Introduction to Sociology courses. Pre- and post-course survey analyses, post-survey student outcomes of a CURE class compared against students enrolled in three non-CURE Introduction to Sociology classes, and a content analysis of end-of-semester papers indicate student knowledge gain in specific topical areas, methodological skills, and major sociology theoretical perspectives. We conclude that UR enhances research- and sociology-related knowledge.

Focused Cardiac Ultrasound Curriculum for Internal Medicine Residents.

Background: Focused cardiac ultrasound (FCU) is a safe and efficient diagnostic intervention for internal medicine physicians. FCU is a highly teachable skill, but is used in routine cardiac assessment in only 20% of surveyed training programs.We developed an FCU curriculum for internal medicine residents and an assessment tool to evaluate the impact of the curriculum on trainee knowledge and confidence. Methods: Internal medicine residents rotating through clinical cardiology services underwent 30 minutes of didactic and 60 minutes of hands-on teaching on acquisition and interpretation of FCU. A 20 item pre and post-curriculum online survey was administered (November 2018-December 2019) to assess confidence and knowledge in FCU. Results: 79 of 116 (68%) residents completed the pre-survey and 50 completed the post-survey, of whom 34 received the curriculum. The mean change in confidence score in those who received versus did not receive the curriculum was 0.99 versus 0.39 (p=0.046) on a 5-point Likert scale. Among 33 residents who had paired pre- and post-surveys the mean change in confidence score was 1.2 versus 0.85 (p<0.001) in those who received versus did not receive the curriculum. The mean increase in knowledge score was 13% versus 7% respectively (p<0.0001). Conclusions: We instituted a novel curriculum for internal medicine residents to gain experience in image acquisition and interpretation. Both confidence and knowledge in FCU improved following the curriculum, indicating that this is a highly teachable skill. Additional analysis of the of the FCU study images will be useful for informing future interventions.

Establishing A Global EBP Tool to Measure Evidence-Based Practice Knowledge: Translating and Testing the French Version of EKAN.

BACKGROUND AND PURPOSE: Government and professional organizations around the world have issued a call for evidence-based practice (EBP), affecting nursing globally. This study aimed to establish a baseline measure of EBP knowledge in French-speaking, baccalaureate nursing students and test the psychometric performance of a newly translated instrument. METHODS: For the cross-sectional, descriptive, correlational study, 103 baccalaureate nursing students from Haiti were included in the analysis. RESULTS: Rasch analysis showed strong evidence of validity (Weighted MS M = 984; Unweighted MS M = 1.01). Individual items showed item reliability (.94). The mean EBP knowledge score was 5.9 (SD = 1.93; Range = 2-10 points). CONCLUSIONS: This study provides initial psychometric evidence for an EBP knowledge measure that can be used as a catalyst for research in 29 French-speaking countries.

Publishing your educational research.

Publishing has always been a part of academic tradition and there is increasing pressure on faculty to publish, even those who carry heavy teaching loads. This article, based on a presentation at the IUBMB 2019 Education Conference session on Publishing in Education, contains suggestions on how to conduct educational research with an eye toward publishing your findings.

When the Clinical Environment Is Closed to Students: The Harsh Reality of COVID-19 and Implications for Colleges of Osteopathic Medicine.

When the World Health Organization (WHO) declared coronavirus disease 2019 (COVID-19) a global health emergency, Colleges of Osteopathic Medicine (COMs) debated the role of medical students during this developing pandemic. Initially, the discussion included whether medical students were essential personnel contributing to meaningful patient care. Many questions arose regarding how COVID-19 would affect medical education and if the changes would be temporary or continue for a significant period of time. Due to the lack of availability of personal protective equipment (PPE) and a decreased focus on clinical education within many healthcare settings, in March the American Association of Colleges of Osteopathic Medicine (AACOM) declared that medical students were not essential personnel and recommended that COMs take a 'pause' and remove students from the clinical environment. This 'pause' would allow COMs time to assess where medical education could continue, to define the critical pieces of clinical education that required a clinical environment and to address how medical students could contribute during the pandemic. The AACOM Clinical Educators group began to meet on a weekly basis during this time so that Clinical Deans from Osteopathic medical schools across the country could collaborate, share ideas, discuss current challenges, and co-create a system to deliver medical education realizing the limitations of in-person clinical training.

Characterization of Use of the Pharmacy Curriculum Outcomes Assessment Across Accredited Colleges of Pharmacy.

Objective. To characterize use of the Pharmacy Curriculum Outcomes Assessment (PCOA) in terms of timing, manner of delivery, and application of the results by accredited colleges of pharmacy. Methods. Accredited pharmacy programs were surveyed regarding PCOA administration, perceived benefits, and practical application of score reports. Survey items were comprised of new items developed from a literature review and items from prior studies. The survey addressed five domains: program demographics, administration, student preparation, use of results, and recommendations to improve the utility of the PCOA. Results. Responses were received from 126 of 139 (91%) surveyed programs. The majority of respondent programs administered PCOA in one session on a single campus. Most indicated PCOA results had limited use for individual student assessment. Almost half reported that results were or could be useful in curriculum review and benchmarking. Considerable variability existed in the preparation and incentives for PCOA performance. Differences in some results were found based on prior PCOA experience and between new vs older programs. Open-ended responses provided suggestions to enhance the application and utility of PCOA. Conclusion. The intended uses of PCOA results, such as for student assessment, curricular review, and programmatic benchmarking, are not being implemented across the academy. Streamlining examination logistics, providing additional examination-related data, and clarifying the purpose of the examination to faculty members and students may increase the utility of PCOA results.

Students' self-assessment of achievement of terminal competency and 4-year trend of student evaluation on outcome-based education.

PURPOSE: This study was designed to allow a student at School of Medicine, Kyungpook National University (KNUSOM) to self-assess how well they had achieved competency and to analyze the differences and trends of the scores by year. Furthermore, students are asked to evaluate the need for curriculum improvement by competency, the tendency of the score is analyzed by year, and the results are reflected in the improvement of the curriculum. METHODS: We conducted a questionnaire survey for fourth-year students of KNUSOM who took medical education classes from 2015 to 2018. Questionnaire items were evaluated on the basis of their current achievement of 30 subordinate competencies of competency and the degree of necessity with respect to revising the curriculum. One-way analysis of variance was performed for the yearly difference analysis. RESULTS: The students' scores on the graduation competency were 2.03 to 4.06. In the yearly difference analysis, there was no significant difference in the average of 30 total competencies, but 16 of the sub-competencies showed significant year-to-year differences. The scores for the 30 graduation competencies were different for each year, but the competencies showing high scores and low scores were found to be similar each year. CONCLUSION: We found that the achievement level of the students was approximately 60% to 70%. We were able to confirm the contents of the education that the students continuously demand. The curriculum trend graphs for each year showed that the students' scores improved when the curriculum was being revised. We found that it is necessary to accept the students' self-evaluation reliable as the students indicated that the contents of the curriculum should be added to the areas where the contents were lacking in the present curriculum.

A multimodal high-value curriculum affects drivers of utilization and performance on the high-value care component of the internal medicine in-training exam.

Background: Teaching the practice of high-value care (HVC) is an increasingly important function of graduate medical education but best practices and long-term outcomes remain unknown. Objective: Whether a multimodal curriculum designed to address specific drivers of low-value care would affect resident attitudes, skills, and performance of HVC as tested by the Internal Medicine In-Training Exam (ITE). Methods: In 2012, we performed a baseline needs assessment among internal medicine residents at a community program regarding drivers of healthcare utilization. We then created a multimodal curriculum with online interactive worksheets, lectures, and faculty buy-in to target specific skills, knowledge, and culture deficiencies. Perceived drivers of care and performance on the Internal Medicine ITE were assessed yearly through 2016. Results: Fourteen of 27 (52%) residents completed the initial needs assessment while the curriculum was eventually seen by at least 24 of 27 (89%). The ITE was taken by every resident every year. Long-term, 3-year follow-up demonstrated persistent improvement in many drivers of utilization (patient requests, reliance on subspecialists, defensive medicine, and academic curiosity) and improvement with sustained high performance on the high-value component of the ITE. Conclusion: A multimodal curriculum targeting specific drivers of low-value care can change culture and lead to sustained improvement in the practice of HVC.

Assessing stakeholder perceptions of the american society for biochemistry and molecular biology accreditation program for baccalaureate degrees.

The American Society for Biochemistry and Molecular Biology (ASBMB) began an accreditation program in 2013. The criteria for accreditation of undergraduate programs include sufficient infrastructure - number and expertise of faculty, physical space and equipment, support for faculty and students - and incorporation of core concepts in the curriculum - structure and function of biomolecules; information storage; energy transfer; and quantitative skills. Students in accredited programs are able to have their degrees ASBMB certified by taking an exam focused on knowledge or skills across the four core concept areas. Members of the accreditation committees administered a survey to key stakeholders in the BMB community: undergraduate programs, both those that have applied for accreditation and those that have not; alumni/ae of accredited programs; graduate and professional programs; and employers. The goals of the study were to gauge the success of the program and determine necessary areas of improvement. The results indicate that the major benefits of applying for accreditation are the impetus to gather data and analysis not generally collected, and access to assessment data via the exam. However, stakeholders outside of the undergraduate community showed little awareness of the accreditation program. Additionally, the application process itself was seen to be very time consuming. This feedback will be used to improve the process and engage in further outreach. © 2018 International Union of Biochemistry and Molecular Biology, 46(5):464-471, 2018.