Using Large Language Models to Support Content Analysis: A Case Study of ChatGPT for Adverse Event Detection.
J Med Internet Res
; 26: e52499, 2024 May 02.
Article
en En
| MEDLINE
| ID: mdl-38696245
ABSTRACT
This study explores the potential of using large language models to assist content analysis by conducting a case study to identify adverse events (AEs) in social media posts. The case study compares ChatGPT's performance with human annotators' in detecting AEs associated with delta-8-tetrahydrocannabinol, a cannabis-derived product. Using the identical instructions given to human annotators, ChatGPT closely approximated human results, with a high degree of agreement noted 94.4% (9436/10,000) for any AE detection (Fleiss κ=0.95) and 99.3% (9931/10,000) for serious AEs (κ=0.96). These findings suggest that ChatGPT has the potential to replicate human annotation accurately and efficiently. The study recognizes possible limitations, including concerns about the generalizability due to ChatGPT's training data, and prompts further research with different models, data sources, and content analysis tasks. The study highlights the promise of large language models for enhancing the efficiency of biomedical research.
Palabras clave
Texto completo:
1
Colección:
01-internacional
Base de datos:
MEDLINE
Asunto principal:
Medios de Comunicación Sociales
Límite:
Humans
Idioma:
En
Revista:
J Med Internet Res
Asunto de la revista:
INFORMATICA MEDICA
Año:
2024
Tipo del documento:
Article
País de afiliación:
Estados Unidos