Designing an evaluation tool for evaluating training programs of medical students in clinical skill training center from consumers' perspective.

INTRODUCTION: The Clinical Skill Training Center (CSTC) is the first environment where third year medical students learn clinical skills after passing basic science. Consumer- based evaluation is one of the ways to improve this center with the consumer. This study was conducted with the aim of preparing a consumer-oriented evaluation tool for CSTC among medical students. METHOD: The study was mixed method. The first phase was qualitative and for providing an evaluation tool. The second phase was for evaluating the tool. At the first phase, after literature review in the Divergent phase, a complete list of problems in the field of CSTC in medicine schools was prepared. In the convergent step, the prepared list was compared with the standards of clinical education and values of scriven. In the second phase it was evaluated by the scientific and authority committee. Validity has been measured by determining CVR and CVI: Index. The face and content validity of the tool was obtained through the approval of a group of specialists. RESULTS: The findings of the research were in the form of 4 questionnaires: clinical instructors, pre-clinical medical students, and interns. All items were designed as a 5-point Likert. The main areas of evaluation included the objectives and content of training courses, implementation of operations, facilities and equipment, and the environment and indoor space. In order to examine the long-term effects, a special evaluation form was designed for intern. CONCLUSION: The tool for consumer evaluation was designed with good reliability and trustworthiness and suitable for use in the CSTC, and its use can improve the effectiveness of clinical education activities.

Competency-Based Medication Administration and Error Reporting.

BACKGROUND: Accuracy is needed with medication administration, a skill that involves rule-based habits and clinical reasoning. This pilot study investigated the use of an evidence-based checklist for accuracy with oral medication administration and error reporting among prelicensure nursing students. Checklist items were anchored in the mnemonic C-MATCH-REASON© (Client, Medication, ADRs, Time, Client History, Route, Expiration date, Amount, Site, Outcomes, Notation). METHOD: Nineteen participants randomly assigned to crossover sequence AB or BA (A: checklist; B: no checklist) practiced simulation scenarios with embedded errors. Nursing faculty used an observation form to track error data. RESULTS: Using the C-MATCH-REASON© checklist compared with not using the checklist supported rule adherence (p = .005), knowledge-based error reduction (p = .011), and total error reduction (p = .010). The null hypothesis was not rejected for errors found (p = .061) nor reported (p = .144), possibly due to sample size. CONCLUSION: C-MATCH-REASON© was effective for error reduction. Study replication with a larger sample is warranted. [J Nurs Educ. 2024;63(5):320-327.].

Determining the psychometric properties of a written test to assess safe dental practice.

INTRODUCTION: Safe practice in medicine and dentistry has been a global priority area in which large knowledge gaps are present.Patient safety strategies aim at preventing unintended damage to patients that can be caused by healthcare practitioners. One of the components of patient safety is safe clinical practice. Patient safety efforts will help in ensuring safe dental practice for early detection and limiting non-preventable errors.A valid and reliable instrument is required to assess the knowledge of dental students regarding patient safety. OBJECTIVE: To determine the psychometric properties of a written test to assess safe dental practice in undergraduate dental students. MATERIAL AND METHODS: A test comprising 42 multiple-choice questions of one-best type was administered to final year students (52) of a private dental college. Items were developed according to National Board of Medical Examiners item writing guidelines. The content of the test was determined in consultation with dental experts (either professor or associate professor). These experts had to assess each item on the test for language clarity as A: clear, B: ambiguous and relevance as 1: essential, 2: useful, not necessary, 3: not essential. Ethical approval was taken from the concerned dental college. Statistical analysis was done in SPSS V.25 in which descriptive analysis, item analysis and Cronbach's alpha were measured. RESULT: The test scores had a reliability (calculated by Cronbach's alpha) of 0.722 before and 0.855 after removing 15 items. CONCLUSION: A reliable and valid test was developed which will help to assess the knowledge of dental students regarding safe dental practice. This can guide medical educationist to develop or improve patient safety curriculum to ensure safe dental practice.

Patient-management problem in the evaluation of intensive medicine.

INTRODUCTION: In intensive care medicine (ICM), the use of Patient-Management Problem (PMP) remains limited and no feedback from students is available. AIM: To compare the feasibility of employing PMP referring to clinical cases (CC) as assessment tools for appraising the knowledge and competencies in ICM students; and to gather the students' perception regarding this experience. METHODS: it was a cross-sectional randomized trial. Were included, external students in the 3rd year of the 2nd cycle of medical studies (3rd-SCMS) during their ICM externship. All the participants underwent two random draws (the 1st one for assessment tool to be started (PMP or CC) and the 2nd for the passage order for PMP. Two PMPs versus two grouped QCMs-CC were prepared and a satisfaction questionnaire was distributed. The main judgment criterion was the effect of each assessment tool on the students' decision-making process. This focused on the relevance of the elements provided by each technique, the implication and the difficulty felt. The secondary endpoint was the scores taken for each tool tested. RESULTS: 20 students were included. All participants had previous experience with PMPs and only nine were familiar with grouped MCQs-CC. PMP scores were 14.9 for the 1st theme and 15.8 for the 2nd theme. The median of the grouped MCQs-CC scores was 14 [12-16] for both. The scores didn't differ between the two techniques. For the 1st theme: the scores were negatively correlated (r=-0.58 and p=0.007). Students felt a better satisfaction for PMP evaluation (p<10-3), the elements provided by PMP were more relevant for decision-making process (p<10-3), the involvement was more felt with PMP (p<10-3) and difficulty was more felt with CCs (p<10-3). The effect of PMP was found to be significant on clinical reasoning (n=36), self-assessment (n=38), problem solving (n=40) and decision making (n=39). Students recommended strongly PMP as a tool of evaluation in ICM (p<10-3). CONCLUSION: scores were comparable between the two tested techniques. The positive perception of students regarding PMP encourages its generalization and teacher training must be strengthened.

Why should Medical Societies increasingly attend to their Specialist Title exams and why should medical professionals obtain it?

Medical societies must maintain high standards of competence and quality when awarding specialist titles, defining the certification criteria, taking into account the needs and realities of the health system and medical practice.

American Dental Education Association Compendium Entrustable Professional Activities Workgroup report.

PURPOSE: Entrustable professional activities (EPAs) are discrete clinical tasks that can be evaluated to help define readiness for independent practice in the health professions and are intended to increase trust in the dental graduate. EPAs provide a framework that bridges competencies to clinical practice. This report describes the work of the American Dental Education Association (ADEA) Compendium EPA Workgroup to develop a list of EPAs for dental education and supportive resources, including specifications and a glossary. METHODS: Preliminary work including literature and resource review, mapping of existing competencies, and review of other health professions' EPAs informed the development of our EPAs list. Workgroup members achieved consensus using a modified Delphi process. A Qualtrics survey using a validated rubric for the assessment of EPAs as described in peer-reviewed literature was used. Dental educators, including academic deans, were surveyed for feedback on the content and format of the EPAs. RESULTS: Based on findings in the literature analysis of existing EPAs and competencies in health professions, a list of EPAs was developed along with a description of specifications. The EPA workgroup (nine members from multiple institutions) used the Delphi process in receiving feedback from various experts. A list of 11 core EPAs was vetted by dental educators including academic deans (n = âˆ¼23), and the process of development was reviewed by EPAs experts outside dental education. A glossary was developed to align language. CONCLUSION: These EPAs define the scope of dental practice. This report represents Phase 1 of the EPA framework development and vetting process. Future directions will include a broader vetting of the EPA list, faculty development, and national standardized technology that support this work to optimize implementation.

Competencies for Those Who Coach Physicians: A Modified Delphi Study.

The rapidly evolving coaching profession has permeated the health care industry and is gaining ground as a viable solution for addressing physician burnout, turnover, and leadership crises that plague the industry. Although various coach credentialing bodies are established, the profession has no standardized competencies for physician coaching as a specialty practice area, creating a market of aspiring coaches with varying degrees of expertise. To address this gap, we employed a modified Delphi approach to arrive at expert consensus on competencies necessary for coaching physicians and physician leaders. Informed by the National Board of Medical Examiners' practice of rapid blueprinting, a group of 11 expert physician coaches generated an initial list of key thematic areas and specific competencies within them. The competency document was then distributed for agreement rating and comment to over 100 stakeholders involved in physician coaching. Our consensus threshold was defined at 70% agreement, and actual responses ranged from 80.5% to 95.6% agreement. Comments were discussed and addressed by 3 members of the original group, resulting in a final model of 129 specific competencies in the following areas: (1) physician-specific coaching, (2) understanding physician and health care context, culture, and career span, (3) coaching theory and science, (4) diversity, equity, inclusion, and other social dynamics, (5) well-being and burnout, and (6) physician leadership. This consensus on physician coaching competencies represents a critical step toward establishing standards that inform coach education, training, and certification programs, as well as guide the selection of coaches and evaluation of coaching in health care settings.

Exploring the Performance of ChatGPT-4 in the Taiwan Audiologist Qualification Examination: Preliminary Observational Study Highlighting the Potential of AI Chatbots in Hearing Care.

Background: Artificial intelligence (AI) chatbots, such as ChatGPT-4, have shown immense potential for application across various aspects of medicine, including medical education, clinical practice, and research. Objective: This study aimed to evaluate the performance of ChatGPT-4 in the 2023 Taiwan Audiologist Qualification Examination, thereby preliminarily exploring the potential utility of AI chatbots in the fields of audiology and hearing care services. Methods: ChatGPT-4 was tasked to provide answers and reasoning for the 2023 Taiwan Audiologist Qualification Examination. The examination encompassed six subjects: (1) basic auditory science, (2) behavioral audiology, (3) electrophysiological audiology, (4) principles and practice of hearing devices, (5) health and rehabilitation of the auditory and balance systems, and (6) auditory and speech communication disorders (including professional ethics). Each subject included 50 multiple-choice questions, with the exception of behavioral audiology, which had 49 questions, amounting to a total of 299 questions. Results: The correct answer rates across the 6 subjects were as follows: 88% for basic auditory science, 63% for behavioral audiology, 58% for electrophysiological audiology, 72% for principles and practice of hearing devices, 80% for health and rehabilitation of the auditory and balance systems, and 86% for auditory and speech communication disorders (including professional ethics). The overall accuracy rate for the 299 questions was 75%, which surpasses the examination's passing criteria of an average 60% accuracy rate across all subjects. A comprehensive review of ChatGPT-4's responses indicated that incorrect answers were predominantly due to information errors. Conclusions: ChatGPT-4 demonstrated a robust performance in the Taiwan Audiologist Qualification Examination, showcasing effective logical reasoning skills. Our results suggest that with enhanced information accuracy, ChatGPT-4's performance could be further improved. This study indicates significant potential for the application of AI chatbots in audiology and hearing care services.

Implementation of a programmatic assessment model in radiation oncology medical physics training.

PURPOSE: In 2019, a formal review and update of the current training program for medical physics residents/registrars in Australasia was conducted. The purpose of this was to ensure the program met current local clinical and technological requirements, to improve standardization of training across Australia and New Zealand and generate a dynamic curriculum and programmatic assessment model. METHODS: A four-phase project was initiated, including a consultant desktop review of the current program and stakeholder consultation. Overarching program outcomes on which to base the training model were developed, with content experts used to update the scientific content. Finally, assessment specialists reviewed a range of assessment models to determine appropriate assessment methods for each learning outcome, creating a model of programmatic assessment. RESULTS: The first phase identified a need for increased standardized assessment incorporating programmatic assessment. Seven clear program outcome statements were generated and used to guide and underpin the new curriculum framework. The curriculum was expanded from the previous version to include emerging technologies, while removing previous duplication. Finally, a range of proposed assessments for learning outcomes in the curriculum were generated into the programmatic assessment model. These new assessment methods were structured to incorporate rubric scoring to provide meaningful feedback. CONCLUSIONS: An updated training program for Radiation Oncology Medial Physics registrars/residents was released in Australasia. Scientific content from a previous program was used as a foundation and revised for currency with the ability to accommodate a dynamic curriculum model. A programmatic model of assessment was created after comprehensive review and consultation. This new model of assessment provides more structured, ongoing assessment throughout the training period. It contains allowances for local bespoke assessment, and guidance for supervisors by the provision of marking templates and rubrics.

Exploring the Performance of ChatGPT Versions 3.5, 4, and 4 With Vision in the Chilean Medical Licensing Examination: Observational Study.

Background: The deployment of OpenAI's ChatGPT-3.5 and its subsequent versions, ChatGPT-4 and ChatGPT-4 With Vision (4V; also known as "GPT-4 Turbo With Vision"), has notably influenced the medical field. Having demonstrated remarkable performance in medical examinations globally, these models show potential for educational applications. However, their effectiveness in non-English contexts, particularly in Chile's medical licensing examinations-a critical step for medical practitioners in Chile-is less explored. This gap highlights the need to evaluate ChatGPT's adaptability to diverse linguistic and cultural contexts. Objective: This study aims to evaluate the performance of ChatGPT versions 3.5, 4, and 4V in the EUNACOM (Examen Único Nacional de Conocimientos de Medicina), a major medical examination in Chile. Methods: Three official practice drills (540 questions) from the University of Chile, mirroring the EUNACOM's structure and difficulty, were used to test ChatGPT versions 3.5, 4, and 4V. The 3 ChatGPT versions were provided 3 attempts for each drill. Responses to questions during each attempt were systematically categorized and analyzed to assess their accuracy rate. Results: All versions of ChatGPT passed the EUNACOM drills. Specifically, versions 4 and 4V outperformed version 3.5, achieving average accuracy rates of 79.32% and 78.83%, respectively, compared to 57.53% for version 3.5 (P<.001). Version 4V, however, did not outperform version 4 (P=.73), despite the additional visual capabilities. We also evaluated ChatGPT's performance in different medical areas of the EUNACOM and found that versions 4 and 4V consistently outperformed version 3.5. Across the different medical areas, version 3.5 displayed the highest accuracy in psychiatry (69.84%), while versions 4 and 4V achieved the highest accuracy in surgery (90.00% and 86.11%, respectively). Versions 3.5 and 4 had the lowest performance in internal medicine (52.74% and 75.62%, respectively), while version 4V had the lowest performance in public health (74.07%). Conclusions: This study reveals ChatGPT's ability to pass the EUNACOM, with distinct proficiencies across versions 3.5, 4, and 4V. Notably, advancements in artificial intelligence (AI) have not significantly led to enhancements in performance on image-based questions. The variations in proficiency across medical fields suggest the need for more nuanced AI training. Additionally, the study underscores the importance of exploring innovative approaches to using AI to augment human cognition and enhance the learning process. Such advancements have the potential to significantly influence medical education, fostering not only knowledge acquisition but also the development of critical thinking and problem-solving skills among health care professionals.

The identification of requirements for competency development during work-integrated learning in healthcare education.

BACKGROUND: Work-integrated learning (WIL) is widely accepted and necessary to attain the essential competencies healthcare students need at their future workplaces. Yet, competency-based education (CBE) remains complex. There often is a focus on daily practice during WIL. Hereby, continuous competency development is at stake. Moreover, the fact that competencies need to continuously develop is often neglected. OBJECTIVES: To ultimately contribute to the optimization of CBE in healthcare education, this study aimed at examining how competency development during WIL in healthcare education could be optimized, before and after graduation. METHODS: Fourteen semi-structured interviews with 16 experts in competency development and WIL were carried out. Eight healthcare disciplines were included namely associate degree nursing, audiology, family medicine, nursing (bachelor), occupational therapy, podiatry, pediatrics, and speech therapy. Moreover, two independent experts outside the healthcare domain were included to broaden the perspectives on competency development. A qualitative research approach was used based on an inductive thematic analysis using Nvivo12© where 'in vivo' codes were clustered as sub-themes and themes. RESULTS: The analysis revealed eight types of requirements for effective and continuous competency development, namely requirements in the context of (1) competency frameworks, (2) reflection and feedback, (3) assessment, (4) the continuity of competency development, (5) mentor involvement, (6) ePortfolios, (7) competency development visualizations, and (8) competency development after graduation. It was noteworthy that certain requirements were fulfilled in one educational program whereas they were absent in another. This emphasizes the large differences in how competence-based education is taking shape in different educational programs and internship contexts. Nevertheless, all educational programs seemed to recognize the importance of ongoing competency development. CONCLUSION: The results of this study indicate that identifying and meeting the requirements for effective and continuous competency development is essential to optimize competency development during practice in healthcare education.

Entrustable professional activities-based objective structured clinical examinations in a pharmacy curriculum.

BACKGROUND: The integration of Objective Structured Clinical Examinations (OSCEs) within the professional pharmacy program, contributes to assessing the readiness of pharmacy students for Advanced Pharmacy Practice Experiences (APPEs) and real-world practice. METHODS: In a study conducted at an Accreditation Council for Pharmacy Education (ACPE)-accredited Doctor of Pharmacy professional degree program, 69 students in their second professional year (P2) were engaged in OSCEs. These comprised 3 stations: best possible medication history, patient education, and healthcare provider communication. These stations were aligned with Entrustable Professional Activities (EPAs) and Ability Statements (AS). The assessment aimed to evaluate pharmacy students' competencies in key areas such as ethical and legal behaviors, general communication skills, and interprofessional collaboration. RESULTS: The formulation of the OSCE stations highlighted the importance of aligning the learning objectives of the different stations with EPAs and AS. The evaluation of students' ethical and legal behaviors, the interprofessional general communication, and collaboration showed average scores of 82.6%, 88.3%, 89.3%, respectively. Student performance on communication-related statements exceeded 80% in all 3 stations. A significant difference (p < 0.0001) was found between the scores of the observer and the SP evaluator in stations 1 and 2 while comparable results (p = 0.426) were shown between the observer and the HCP evaluator in station 3. Additionally, a discrepancy among the observers' assessments was detected across the 3 stations. The study shed light on challenges encountered during OSCEs implementation, including faculty involvement, resource constraints, and the necessity for consistent evaluation criteria. CONCLUSIONS: This study highlights the importance of refining OSCEs to align with EPAs and AS, ensuring a reliable assessment of pharmacy students' clinical competencies and their preparedness for professional practice. It emphasizes the ongoing efforts needed to enhance the structure, content, and delivery of OSCEs in pharmacy education. The findings serve as a catalyst for addressing identified challenges and advancing the effectiveness of OSCEs in accurately evaluating students' clinical readiness.

Point-of-care ultrasound in pediatric anesthesiology: considerations for training and credentialing.

PURPOSE OF REVIEW: To discuss considerations surrounding the use of point-of-care ultrasound (POCUS) in pediatric anesthesiology. RECENT FINDINGS: POCUS is an indispensable tool in various medical specialties, including pediatric anesthesiology. Credentialing for POCUS should be considered to ensure that practitioners are able to acquire images, interpret them correctly, and use ultrasound to guide procedures safely and effectively. In the absence of formal guidelines for anesthesiology, current practice and oversight varies by institution. In this review, we will explore the significance of POCUS in pediatric anesthesiology, discuss credentialing, and compare the specific requirements and challenges currently associated with using POCUS in pediatric anesthesia. SUMMARY: Point-of-care ultrasound is being utilized by the pediatric anesthesiologist and has the potential to improve patient assessment, procedure guidance, and decision-making. Guidelines increase standardization and quality assurance procedures help maintain high-quality data. Credentialing standards for POCUS in pediatric anesthesiology are essential to ensure that practitioners have the necessary skills and knowledge to use this technology effectively and safely. Currently, there are no national pediatric POCUS guidelines to base credentialing processes on for pediatric anesthesia practices. Further work directed at establishing pediatric-specific curriculum goals and competency standards are needed to train current and future pediatric anesthesia providers and increase overall acceptance of POCUS use.

Implementation of standard setting for high-stakes objective structured clinical examinations.

BACKGROUND AND PURPOSE: To describe one institution's approach to transformation of high-stakes objective structure clinical examinations (OSCEs) from norm-referenced to criterion-referenced standards setting and to evaluate the impact of these changes on OSCE performance and pass rates. EDUCATIONAL ACTIVITY AND SETTING: The OSCE writing team at the college selected a modified Angoff method appropriate for high-stakes assessments to replace the two standard deviation method previously used. Each member of the OSCE writing team independently reviewed the analytical checklist and calculated a passing score for active stations on OSCEs. Then the group met to determine a final pass score for each station. The team also determined critical cut points for each station, when indicated. After administration of the OSCEs, scores, pass rates, and need for remediation were compared to the previous norm-referenced method. Descriptive statistics were used to summarize the data. FINDINGS: OSCE scores remained relatively unchanged when switched to a criterion-referenced method, but the number of remediators increased up to 2.6 fold. In the first year, the average score increased from 86.8% to 91.7% while the remediation rate increased from 2.8% to 7.4%. In the third year, the average increased from 90.9% to 92% while the remediation rate increased from 6% to 15.6%. Likewise, the fourth-year average increased from 84.9% to 87.5% while the remediation rate increased from 4.4% to 9%. SUMMARY: Transition to a modified Angoff method did not impact average OSCE score but did increase the number of remediations.

Core Palliative Care Competencies for Undergraduate Nursing Education: International Multisite Research Using Online Nominal Group Technique.

Background: Nurses should have appropriate education and required competencies to provide high-quality palliative care. The aim of this international multisite study was to list and evaluate core palliative care competencies that European nurses need to achieve in their education to provide palliative care. Methods: The Nominal Group Technique (NGT) was used as a data collection method. NGT meetings were organized in four European countries. Targeted groups of palliative care professionals with diverse contextual and professional backgrounds participated in the NGTs. The research question was: "What are the core competencies in palliative care that need to be achieved during undergraduate nursing education?" Data analysis was done in two stages: grouping the top 10 answers based on similarities and thematic synthesis based on all the ideas produced during the NGTs. Results: Palliative care core competencies based on the research were (1) competence in the characteristics of palliative care; (2) competence in decision-making and enabling palliative care; (3) symptom management competence in palliative care; (4) competence in holistic support in palliative care; (5) active person- and family-centered communication competence in palliative care; (6) competence in empathy in palliative care; (7) spiritual competence in palliative care; (8) competence in ethical and legal issues in palliative care; (9) teamwork competence in palliative care; and (10) self-awareness and self-reflection competence in palliative care. Conclusions: It was possible to find differences and similarities in the top 10 palliative care core competencies from different countries. Thematic synthesis of all the data showed that there were various competencies needed for nursing students to provide quality palliative care.

Utility of RAND/UCLA appropriateness method in validating multiple-choice questions on ECG.

OBJECTIVES: This study aimed to investigate the utility of the RAND/UCLA appropriateness method (RAM) in validating expert consensus-based multiple-choice questions (MCQs) on electrocardiogram (ECG). METHODS: According to the RAM user's manual, nine panelists comprising various experts who routinely handle ECGs were asked to reach a consensus in three phases: a preparatory phase (round 0), an online test phase (round 1), and a face-to-face expert panel meeting (round 2). In round 0, the objectives and future timeline of the study were elucidated to the nine expert panelists with a summary of relevant literature. In round 1, 100 ECG questions prepared by two skilled cardiologists were answered, and the success rate was calculated by dividing the number of correct answers by 9. Furthermore, the questions were stratified into "Appropriate," "Discussion," or "Inappropriate" according to the median score and interquartile range (IQR) of appropriateness rating by nine panelists. In round 2, the validity of the 100 ECG questions was discussed in an expert panel meeting according to the results of round 1 and finally reassessed as "Appropriate," "Candidate," "Revision," and "Defer." RESULTS: In round 1 results, the average success rate of the nine experts was 0.89. Using the median score and IQR, 54 questions were classified as " Discussion." In the expert panel meeting in round 2, 23% of the original 100 questions was ultimately deemed inappropriate, although they had been prepared by two skilled cardiologists. Most of the 46 questions categorized as "Appropriate" using the median score and IQR in round 1 were considered "Appropriate" even after round 2 (44/46, 95.7%). CONCLUSIONS: The use of the median score and IQR allowed for a more objective determination of question validity. The RAM may help select appropriate questions, contributing to the preparation of higher-quality tests.