Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis.

BACKGROUND: Over the past 2 years, researchers have used various medical licensing examinations to test whether ChatGPT (OpenAI) possesses accurate medical knowledge. The performance of each version of ChatGPT on the medical licensing examination in multiple environments showed remarkable differences. At this stage, there is still a lack of a comprehensive understanding of the variability in ChatGPT's performance on different medical licensing examinations. OBJECTIVE: In this study, we reviewed all studies on ChatGPT performance in medical licensing examinations up to March 2024. This review aims to contribute to the evolving discourse on artificial intelligence (AI) in medical education by providing a comprehensive analysis of the performance of ChatGPT in various environments. The insights gained from this systematic review will guide educators, policymakers, and technical experts to effectively and judiciously use AI in medical education. METHODS: We searched the literature published between January 1, 2022, and March 29, 2024, by searching query strings in Web of Science, PubMed, and Scopus. Two authors screened the literature according to the inclusion and exclusion criteria, extracted data, and independently assessed the quality of the literature concerning Quality Assessment of Diagnostic Accuracy Studies-2. We conducted both qualitative and quantitative analyses. RESULTS: A total of 45 studies on the performance of different versions of ChatGPT in medical licensing examinations were included in this study. GPT-4 achieved an overall accuracy rate of 81% (95% CI 78-84; P<.01), significantly surpassing the 58% (95% CI 53-63; P<.01) accuracy rate of GPT-3.5. GPT-4 passed the medical examinations in 26 of 29 cases, outperforming the average scores of medical students in 13 of 17 cases. Translating the examination questions into English improved GPT-3.5's performance but did not affect GPT-4. GPT-3.5 showed no difference in performance between examinations from English-speaking and non-English-speaking countries (P=.72), but GPT-4 performed better on examinations from English-speaking countries significantly (P=.02). Any type of prompt could significantly improve GPT-3.5's (P=.03) and GPT-4's (P<.01) performance. GPT-3.5 performed better on short-text questions than on long-text questions. The difficulty of the questions affected the performance of GPT-3.5 and GPT-4. In image-based multiple-choice questions (MCQs), ChatGPT's accuracy rate ranges from 13.1% to 100%. ChatGPT performed significantly worse on open-ended questions than on MCQs. CONCLUSIONS: GPT-4 demonstrates considerable potential for future use in medical education. However, due to its insufficient accuracy, inconsistent performance, and the challenges posed by differing medical policies and knowledge across countries, GPT-4 is not yet suitable for use in medical education. TRIAL REGISTRATION: PROSPERO CRD42024506687; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=506687.

Encouraging COVID-19 vaccination by focusing on anticipated affect: A scoping review.

Objective: This study reviewed studies of the anticipated affect related with COVID-19 vaccination to understand gaps in currently available studies and practice implications. Methods: We systematically searched MEDLINE, CINAHL, and other multiple databases for English language articles of studies that investigated COVID-19 vaccination related anticipated affects. Results: We identified seventeen studies. Thirteen studies focused anticipated regret from inaction (i.e., not vaccinated). Other studies focused anticipated regret from action (i.e., vaccinated), guilt from inaction, pride from action, and positive feelings from action. Eleven studies showed that anticipated regret from inaction was significantly associated with COVID-19 vaccination behavior or intention. Three of the 11 studies showed that anticipated regret from inaction was more strongly associated with vaccination behavior or intention than cognitive belief. Conclusion: Most studies showed that positive associations between anticipated regret and COVID-19 vaccination outcomes. The use of messages that target cognitive beliefs as well as those that appeal to anticipated affect may be effective to promote COVID-19 vaccination. However, most studies employed a cross-sectional design and examined negative affect. Future studies should adopt an experimental design as well as examine positive affect.

Development of the Japanese Version of Pregnancy-Related Anxiety Questionnaire-Revised-2: Measurement and Psychometric Properties.

Tools to evaluate pregnancy-specific anxiety are lacking in Japan. This study aimed to develop a Japanese version of the Pregnancy-Related Anxiety Questionnaire-Revised-2. After scale translation and cognitive interviews, we conducted a cross-sectional study among 120 ≥18-year-old, singleton (pregnant with one baby) Japanese women before 15 weeks of pregnancy, recruited from four facilities. A total of 112 women completed the questionnaires. We tested the internal consistency, measurement error and reliability, structural validity, measurement invariance across nulliparous and parous women, construct validity by calculating omega, standard error of measurement (SEM), intraclass correlation coefficient (ICC), confirmatory factor analysis (CFA), multigroup CFA, multitrait-scaling analysis, correlational analyses with other measurements, and t-test to compare nulliparous and parous groups. Omega was 0.90 for the total score. SEM was 3.4 and ICC was 0.76. The CFA revealed an optimal fit for the three-factor model based on the original scale. Multigroup CFA supported measurement invariance across the nulliparous and parous groups, and multitrait-scaling analysis revealed 100% scaling success. The correlation coefficients with other scales of childbirth anxiety and general anxiety were 0.70 and 0.24. The mean total score of the nulliparous women was higher than that of the parous women (34.5 vs. 30.3, p = 0.001). Therefore, the scale was determined to have good validity and reliability.

Translation, Cross-Cultural Adaptation, and Validation of the Japanese Version of the Patient Education Materials Assessment Tool (PEMAT).

BACKGROUND: The Patient Education Materials Assessment Tool (PEMAT) systematically evaluates the understandability and actionability of patient education materials. This study aimed to develop a Japanese version of PEMAT and verify its reliability and validity. METHODS: After assessing content validation, experts scored healthcare-related leaflets and videos according to PEMAT to verify inter-rater reliability. In validation testing with laypeople, the high-scoring material group (n = 800) was presented with materials that received high ratings on PEMAT, and the low-scoring material group (n = 799) with materials that received low ratings. Both groups responded to the understandability and actionability of the materials and perceived self-efficacy for the recommended actions. RESULTS: The Japanese version of PEMAT showed strong inter-rater reliability (PEMAT-P: % agreement = 87.3, Gwet's AC1 = 0.83. PEMAT-A/V: % agreement = 85.7, Gwet's AC1 = 0.80). The high-scoring material group had significantly higher scores for understandability and actionability than the low-scoring material group (PEMAT-P: understandability 6.53 vs. 5.96, p < 0.001; actionability 6.04 vs. 5.49, p < 0.001; PEMAT-A/V: understandability 7.65 vs. 6.76, p < 0.001; actionability 7.40 vs. 6.36, p < 0.001). Perceived self-efficacy increased more in the high-scoring material group than in the low-scoring material group. CONCLUSIONS: Our study showed that materials rated highly on Japanese version of PEMAT were also easy for laypeople to understand and action.

Development of the Japanese Version of the Infertility Stigma Scale: Examination of Its Reliability and Validity.

The stigma of infertility negatively affects health, resulting in mental distress and poor quality of life. An appropriate scale is essential to examine the stigma experienced by infertile women and provide adequate interventions. Therefore, we developed a Japanese version of the Infertility Stigma Scale (ISS). After examining the content validity of this scale, we conducted an online survey of women undergoing fertility treatment to test the scale's structural validity, internal consistency, intra-rater reliability, known-groups validity, convergent validity, and discriminant validity. A total of 254 participants were included in the analysis. The results of confirmatory factor analysis of four factors based on the original scale revealed the optimal fit. Cronbach's alpha was 0.95 for the total score. Concerning test−retest analysis, the total score of the ISS and subscale had a high Spearman correlation coefficient (ρ = 0.61−0.88, p < 0.001). For convergent validity, the association between the ISS and social support, self-esteem, and family functioning was significantly negatively correlated. The results of the multitrait scaling analysis scale showed that correlations of all items exceeded 0.40, and scaling errors (7/81, 8.6%) were few. The Japanese version of the ISS was confirmed to have acceptable reliability and validity.

Japanese newspaper advertisements for dietary supplements before and after COVID-19: a content analysis.

OBJECTIVES: Public health institutions have alerted consumers about advertising for dietary supplements with false claims of preventing or treating COVID-19. We quantitatively and qualitatively examined newspaper advertisements for dietary supplements before and after the COVID-19 spread. DESIGN: Content analysis. PARTICIPANTS: We analysed advertisements for dietary supplements in two major Japanese newspapers in February-July 2019 and February-July 2020. Our analysis covered 2167 advertisements. RESULTS: The number of advertisements for dietary supplements that claimed to be effective in infection prevention (p=0.009) and improving joint (p=0.002) and digestive functions (p=0.002) significantly increased after the spread of COVID-19 compared with before. Dietary supplements that claimed to be effective in preventing infection were advertised in combination with recommendations for gargling and handwashing. Such terms as 'defence' and 'prevent' were used to promote the preventive effect. CONCLUSIONS: False and misleading claims in advertising for dietary supplements may result in consumer harm, such as overdosing and failure to take preventive behaviour. While the pandemic continues, there will be an increasing need for disseminating accessible information about the appropriate use of dietary supplements, consumer education and warnings to manufacturers.

Changes in Anxiety and Stress Among Pregnant Women During the COVID-19 Pandemic: Content Analysis of a Japanese Social Question-and-Answer Website.

BACKGROUND: The changing pattern of anxiety and stress experienced by pregnant women during the COVID-19 pandemic is unknown. OBJECTIVE: We aimed to examine the sources of anxiety and stress in pregnant women in Japan during the COVID-19 pandemic. METHODS: We performed content analysis of 1000 questions posted on the largest social website in Japan (Yahoo! Chiebukuro) from January 1 to May 25, 2020 (end date of the national state of emergency). The Gwet AC1 coefficient was used to verify interrater reliability. RESULTS: A total 12 categories were identified. Throughout the study period, anxiety related to going outdoors appeared most frequent, followed by anxiety regarding employment and infection among family and friends. Following the declaration of the state of national emergency at the peak of the infection, infection-related anxiety decreased, whereas anxiety about social support and mood disorders increased. Stress regarding relationships appeared frequent throughout the pandemic. CONCLUSIONS: The sources of anxiety and stress in pregnant women in Japan changed during the pandemic. Our results suggest the need for rapid communications in the early phase of a pandemic as well as long-term psychosocial support to provide optimal support to pregnant women in Japan. Health care professionals should understand the changing pattern of requirements among pregnant women.

Fermented and nonfermented soy foods and the risk of breast cancer in a Japanese population-based cohort study.

BACKGROUND: Although preclinical studies suggest that fermented soy foods may have a protective effect against breast cancer, no prospective cohort studies have examined this association. OBJECTIVE: Our study examined the association between fermented and nonfermented soy food intake and breast cancer risk using a population-based prospective cohort study in Japan. METHODS: We included a total of 47,614 women aged 45-74 years in an analysis of the Japan Public Health Center-based Prospective Study (JPHC Study). A validated food frequency questionnaire (FFQ) was used for the assessment of dietary intake. Breast cancer incidence was analyzed by multivariate Cox proportional hazards regression models. RESULTS: During an average of 15.5 years of follow-up, 825 breast cancer cases were newly identified. We found no association of intake of soy foods with breast cancer risk, regardless of fermentation, with multivariate hazard ratios (HRs) and 95% confidence intervals (CIs) for the highest quartiles of fermented and nonfermented soy food intake of 0.94 (0.67, 1.32) and 1.15 (0.85, 1.57) compared with the lowest quartile (p for trend = 0.305 and 0.393). Unlike nonfermented soy, higher intake of fermented soy foods was associated with a significant decrease in the risk of nonlocalized breast cancer. The HR and 95% CI in the highest compared to lowest intake category of fermented soy foods was 0.53 (0.28, 0.99) versus nonfermented soy foods 0.85 (0.51, 1.42) (p for trend = 0.026 and 0.797). CONCLUSIONS: Our analyses showed that fermented soy foods had no association with overall breast cancer but may be associated with decreased risk of nonlocalized breast cancer.