Performance of ChatGPT-4 in answering questions from the Brazilian National Examination for Medical Degree Revalidation.
Rev Assoc Med Bras (1992)
; 69(10): e20230848, 2023.
Article
en En
| MEDLINE
| ID: mdl-37792871
ABSTRACT
OBJECTIVE:
The aim of this study was to evaluate the performance of ChatGPT-4.0 in answering the 2022 Brazilian National Examination for Medical Degree Revalidation (Revalida) and as a tool to provide feedback on the quality of the examination.METHODS:
A total of two independent physicians entered all examination questions into ChatGPT-4.0. After comparing the outputs with the test solutions, they classified the large language model answers as adequate, inadequate, or indeterminate. In cases of disagreement, they adjudicated and achieved a consensus decision on the ChatGPT accuracy. The performance across medical themes and nullified questions was compared using chi-square statistical analysis.RESULTS:
In the Revalida examination, ChatGPT-4.0 answered 71 (87.7%) questions correctly and 10 (12.3%) incorrectly. There was no statistically significant difference in the proportions of correct answers among different medical themes (p=0.4886). The artificial intelligence model had a lower accuracy of 71.4% in nullified questions, with no statistical difference (p=0.241) between non-nullified and nullified groups.CONCLUSION:
ChatGPT-4.0 showed satisfactory performance for the 2022 Brazilian National Examination for Medical Degree Revalidation. The large language model exhibited worse performance on subjective questions and public healthcare themes. The results of this study suggested that the overall quality of the Revalida examination questions is satisfactory and corroborates the nullified questions.
Texto completo:
1
Colección:
01-internacional
Base de datos:
MEDLINE
Asunto principal:
Inteligencia Artificial
/
Personal de Salud
Tipo de estudio:
Prognostic_studies
Límite:
Humans
País/Región como asunto:
America do sul
/
Brasil
Idioma:
En
Revista:
Rev Assoc Med Bras (1992)
Año:
2023
Tipo del documento:
Article
País de afiliación:
Brasil
Pais de publicación:
BR
/
BRASIL
/
BRASILE
/
BRAZIL
/
BRESIL