Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis.

BACKGROUND: ChatGPT, a recently developed artificial intelligence (AI) chatbot, has demonstrated improved performance in examinations in the medical field. However, thus far, an overall evaluation of the potential of ChatGPT models (ChatGPT-3.5 and GPT-4) in a variety of national health licensing examinations is lacking. This study aimed to provide a comprehensive assessment of the ChatGPT models' performance in national licensing examinations for medical, pharmacy, dentistry, and nursing research through a meta-analysis. METHODS: Following the PRISMA protocol, full-text articles from MEDLINE/PubMed, EMBASE, ERIC, Cochrane Library, Web of Science, and key journals were reviewed from the time of ChatGPT's introduction to February 27, 2024. Studies were eligible if they evaluated the performance of a ChatGPT model (ChatGPT-3.5 or GPT-4); related to national licensing examinations in the fields of medicine, pharmacy, dentistry, or nursing; involved multiple-choice questions; and provided data that enabled the calculation of effect size. Two reviewers independently completed data extraction, coding, and quality assessment. The JBI Critical Appraisal Tools were used to assess the quality of the selected articles. Overall effect size and 95% confidence intervals [CIs] were calculated using a random-effects model. RESULTS: A total of 23 studies were considered for this review, which evaluated the accuracy of four types of national licensing examinations. The selected articles were in the fields of medicine (n = 17), pharmacy (n = 3), nursing (n = 2), and dentistry (n = 1). They reported varying accuracy levels, ranging from 36 to 77% for ChatGPT-3.5 and 64.4-100% for GPT-4. The overall effect size for the percentage of accuracy was 70.1% (95% CI, 65-74.8%), which was statistically significant (p < 0.001). Subgroup analyses revealed that GPT-4 demonstrated significantly higher accuracy in providing correct responses than its earlier version, ChatGPT-3.5. Additionally, in the context of health licensing examinations, the ChatGPT models exhibited greater proficiency in the following order: pharmacy, medicine, dentistry, and nursing. However, the lack of a broader set of questions, including open-ended and scenario-based questions, and significant heterogeneity were limitations of this meta-analysis. CONCLUSIONS: This study sheds light on the accuracy of ChatGPT models in four national health licensing examinations across various countries and provides a practical basis and theoretical support for future research. Further studies are needed to explore their utilization in medical and health education by including a broader and more diverse range of questions, along with more advanced versions of AI chatbots.

Factors Affecting Leadership in Pharmacy Students: A Nationwide Survey.

Purpose: Leadership is increasingly becoming a priority in the forms of higher education associated with the Fourth Industrial Revolution; however, few studies have examined it in the context of pharmacy education. This study investigated the levels of communication competence, critical thinking disposition, problem-solving ability, and leadership of pharmacy students, and identified factors related to leadership. Methods: This study was conducted using a nationwide cross-sectional online survey with a self-administered questionnaire. A total of 416 third-to-sixth-year pharmacy students from all 35 pharmacy schools in South Korea, completed the survey from September 1 to 15, 2019. Data were analyzed using descriptive statistics, t-tests, Pearson's correlations, and multiple regression analysis. We verified construct validity by performing an exploratory factor analysis. Results: The pharmacy students revealed a moderate level of communication skills, critical thinking disposition, problem-solving ability, and leadership. Significant positive correlations were found among communication competence (r=0.724, p<0.001), critical thinking disposition (r=0.615, p<0.001), problem-solving ability (r=0.599, p<0.001), and leadership. After adjustment for control variables, communication competence (ß=0.319, p<0.001) was found to be the most significant predictor of leadership (F=104.12, p<0.001, R2=0.756). Conclusion: This study showed that the students' leadership was influenced by their competence in communicating effectively, thinking critically, and solving problems. Pharmacy educators should acknowledge and implement innovative curriculum and assessment approaches in preparing pharmacy students for their professional paths.

Novel innovative computer-based test (Inno-CBT) item types for national licensing examinations for health care professionals.

BACKGROUND: An effective test mechanism to evaluate clinical knowledge and skills of the entry-level healthcare professionals is important for providing clinical competency and improving patient care. This study aimed to develop novel, innovative computer-based test (Inno-CBT) item types for application in the national examination of Korean healthcare professionals. METHODS: This exploratory study was conducted from May 2021 to March 2022 by a team of faculty members from pharmacy schools in South Korea. A literature search using PubMed, Google Scholar, RISS, Web of Science, and KoreaMed was performed. Forum presentations, media articles, and previous reports by the Korea Health Personnel Licensing Examination Institute (KHPLEI) were included. Workshops were held, information and ideas were collected and conceptualized, and item types were designed, drafted, and refined. By repeating this process, the Inno-CBT item types were finalized. RESULTS: Forty-one Inno-CBT item types with 28 subtypes were developed. New digital technologies, such as a reactive responsive media interface, an animation insertion, multimedia embedding, and network surfing, were utilized in these novel types. It was anticipated that these Inno-CBT item types would effectively measure abilities in healthcare knowledge, problem-solving skills, and professional behaviors. Some potential barriers to implementing the Inno-CBT item types include item difficulty, operational unfamiliarity, complexity in scoring protocols, and network security. CONCLUSIONS: A variety of styles of novel Inno-CBT item types were developed to evaluate the multifaceted and in-depth professional abilities required for healthcare professionals. Prior to implementing these item types in the national examination, item validation and technical support should be conducted.

The influence of a patient counseling training session on pharmacy students' self-perceived communication skills, confidence levels, and attitudes about communication skills training.

BACKGROUND: The ability to communicate effectively is an essential skill for a pharmacist. However, the curricula of most pharmacy schools in South Korea do not include communication skills training (CST). This study aims to evaluate the effects of CST in pharmacy education. METHODS: This study was a comparison of pre- and post-intervention surveys completed by sixty fifth-year pharmacy students who participated in communication skills and patient counseling training during the spring 2017 semester. The students were asked to respond to 49 questions addressing 4 self-assessment categories: communication skills (24), attitudes (19), and confidence levels (2) at the beginning and end of the CST, and their perception of CST (4) after completing the course. The training session included lectures, small group work, role play, videos, and performance feedback by a tutor. Data were analyzed using the paired t-test with Bonferroni's correction for multiple comparisons. The open-ended questions were analyzed using inductive content analysis. RESULTS: The pharmacy students' self-assessment of their communication skills, attitudes toward the communication course, and confidence levels showed significant improvement after the CST. Most students (96.7%) indicated the necessity of a pharmacy communication curriculum. They responded that CST is helpful for effective communication with patients (33.3%) and other healthcare professionals (31.7%). Role-playing was reported as the most preferred learning method (58.3%). CONCLUSIONS: CST significantly impacted pharmacy students' skills, attitudes, and confidence levels related to communication skills and patient counseling. These findings indicate that communications training should be included in the regular curriculum of pharmacy schools.

Nutrition and Clinical Outcomes of Nutrition Support in Multidisciplinary Team for Critically Ill Patients.

BACKGROUND: To examine the outcomes of the implementation of a multidisciplinary nutrition support team (NST) that included a pharmacist for critically ill patients in the intensive care unit (ICU). METHODS: Data were retrospectively collected from electronic medical records and compared between the pre-NST group (n = 73) and post-NST group (n = 75). Patients were included if they received enteral or parenteral nutrition support for at least 72 hours in the ICU of an approximately 900-bed, top-tier university medical center. The percentage of goal kcal, the percentage of goal protein, serum albumin level, total lymphocyte count, C-reactive protein, duration of mechanical ventilation use, hospital length of stay (LOS), ICU LOS, and mortality were evaluated. RESULTS: There were significant differences in the percentage of goal kcal (66.9% ± 25.9% vs 86.2% ± 27.5%; P < 0.001) and the percentage of goal protein (67.0% ± 29.9% vs 81.7% ± 30.7%; P < 0.05) between the pre-NST and post-NST groups. A higher percentage of goal kcal was associated with a significant decrease in the ICU LOS (P < 0.05) and hospital LOS (P < 0.05). The percentage of goal kcal was associated with mortality rate (odd ratio, 0.977; 95% confidential interval, 0.959-0.996; P = 0.016). CONCLUSION: Implementation of a multidisciplinary NST service for critically ill patients was associated with higher percentages of goal kcal and goal protein, as well as a shortened duration of mechanical ventilation use. Higher energy supply was associated with a reduced mortality rate.

The effect of communication skills training on patient-pharmacist communication in pharmacy education: a meta-analysis.

Communication skills in pharmacy education and practice are increasingly regarded as a crucial component. However, thus far, estimating of the overall communication skills training (CST) effects in a variety of outcomes is lacking. The aim of this study was to synthesize the effects of CST in pharmacy education by performing a meta-analysis of CST studies. We searched MEDLINE, EMBASE, ERIC, CINAHL, PsycINFO, Cochrane Library, Web of Science, Communication and Mass Media Complete (CMMC), key journals, and bibliographic databases. The effect sizes (ESs) were extracted and pooled in random effects meta-analyses. We assessed the quality of the study using the Medical Education Research Study Quality Instrument (MERSQI). From 34,737 articles, 9 studies were included in this meta-analysis. The overall effect size for CST was 0.611 (95% CI 0.327-0.895), and it was statistically significant (p = 0.000). We found based on the subgroup analyses that CST has a large effect size when it used stand-alone courses, lecture-lab based courses, video recordings, feedback, training for 2 or more semesters, hours per week ≥5 h and external assessments. For the CST effect, the effect sizes were ranked in order of confidence, knowledge, skills, and attitudes. The result of the meta-regression is that the total number of attendees is significantly negatively correlated with the effect sizes of the CST. The findings of the present meta-analysis provide evidence that CST in pharmacy education may act as an efficient way to improve the communication competency of students, and it may serve as a guide for pharmacy educators.