Comparative accuracy of ChatGPT-4, Microsoft Copilot and Google Gemini in the Italian entrance test for healthcare sciences degrees: a cross-sectional study.

BACKGROUND: Artificial intelligence (AI) chatbots are emerging educational tools for students in healthcare science. However, assessing their accuracy is essential prior to adoption in educational settings. This study aimed to assess the accuracy of predicting the correct answers from three AI chatbots (ChatGPT-4, Microsoft Copilot and Google Gemini) in the Italian entrance standardized examination test of healthcare science degrees (CINECA test). Secondarily, we assessed the narrative coherence of the AI chatbots' responses (i.e., text output) based on three qualitative metrics: the logical rationale behind the chosen answer, the presence of information internal to the question, and presence of information external to the question. METHODS: An observational cross-sectional design was performed in September of 2023. Accuracy of the three chatbots was evaluated for the CINECA test, where questions were formatted using a multiple-choice structure with a single best answer. The outcome is binary (correct or incorrect). Chi-squared test and a post hoc analysis with Bonferroni correction assessed differences among chatbots performance in accuracy. A p-value of < 0.05 was considered statistically significant. A sensitivity analysis was performed, excluding answers that were not applicable (e.g., images). Narrative coherence was analyzed by absolute and relative frequencies of correct answers and errors. RESULTS: Overall, of the 820 CINECA multiple-choice questions inputted into all chatbots, 20 questions were not imported in ChatGPT-4 (n = 808) and Google Gemini (n = 808) due to technical limitations. We found statistically significant differences in the ChatGPT-4 vs Google Gemini and Microsoft Copilot vs Google Gemini comparisons (p-value < 0.001). The narrative coherence of AI chatbots revealed "Logical reasoning" as the prevalent correct answer (n = 622, 81.5%) and "Logical error" as the prevalent incorrect answer (n = 40, 88.9%). CONCLUSIONS: Our main findings reveal that: (A) AI chatbots performed well; (B) ChatGPT-4 and Microsoft Copilot performed better than Google Gemini; and (C) their narrative coherence is primarily logical. Although AI chatbots showed promising accuracy in predicting the correct answer in the Italian entrance university standardized examination test, we encourage candidates to cautiously incorporate this new technology to supplement their learning rather than a primary resource. TRIAL REGISTRATION: Not required.

Safety and Tolerability of COVID-19 Vaccine in Mast Cell Disorders Real-Life Data from a Single Centre in Italy.

Background In the past three years, COVID-19 has had a significant impact on the healthcare systems and people's safety worldwide. Mass vaccinations dramatically improved the health and economic damage caused by SARS-CoV-2. However, the safety of COVID-19 vaccines in patients at high risk of allergic reactions still has many unmet needs that should be clarified. Material and methods A retrospective, single-centre study was performed by collecting demographic and clinical data of patients with Mast Cell Disorders (MCDs) to evaluate the safety and tolerability of COVID-19 vaccinations. Moreover, any changes in the natural history of the underlying disease following the vaccine have been evaluated. Results This study included 66 patients affected with MCDs. Out of them, 52 (78.8%) received a COVID-19 vaccination and 41 (78.8%) completed the vaccination course. Premedication came first in 86.6% of our patients. A total of seven (4.5%) patients complained about an immediate reaction and two (1.3%) had a late reaction. Worsening of MCD history was observed in a single patient. Conclusions Despite the overall high risk of allergic reactions, our study did not reveal any increased risk for SARS-CoV-2 allergic reactions in MCD patients, thus supporting the recommendation in favour of the SARS-CoV-2 vaccination. However, due to the potentially increased rate of anaphylactic reactions, MCD patients should receive vaccine premedication and should be treated in a hospital setting after an allergological specialistic evaluation.