Can ChatGPT pass the MRCP (UK) written examinations? Analysis of performance and errors using a clinical decision-reasoning framework.

OBJECTIVE: Large language models (LLMs) such as ChatGPT are being developed for use in research, medical education and clinical decision systems. However, as their usage increases, LLMs face ongoing regulatory concerns. This study aims to analyse ChatGPT's performance on a postgraduate examination to identify areas of strength and weakness, which may provide further insight into their role in healthcare. DESIGN: We evaluated the performance of ChatGPT 4 (24 May 2023 version) on official MRCP (Membership of the Royal College of Physicians) parts 1 and 2 written examination practice questions. Statistical analysis was performed using Python. Spearman rank correlation assessed the relationship between the probability of correctly answering a question and two variables: question difficulty and question length. Incorrectly answered questions were analysed further using a clinical reasoning framework to assess the errors made. SETTING: Online using ChatGPT web interface. PRIMARY AND SECONDARY OUTCOME MEASURES: Primary outcome was the score (percentage questions correct) in the MRCP postgraduate written examinations. Secondary outcomes were qualitative categorisation of errors using a clinical decision-making framework. RESULTS: ChatGPT achieved accuracy rates of 86.3% (part 1) and 70.3% (part 2). Weak but significant correlations were found between ChatGPT's accuracy and both just-passing rates in part 2 (r=0.34, p=0.0001) and question length in part 1 (r=-0.19, p=0.008). Eight types of error were identified, with the most frequent being factual errors, context errors and omission errors. CONCLUSION: ChatGPT performance greatly exceeded the passing mark for both exams. Multiple choice examinations provide a benchmark for LLM performance which is comparable to human demonstrations of knowledge, while also highlighting the errors LLMs make. Understanding the reasons behind ChatGPT's errors allows us to develop strategies to prevent them in medical devices that incorporate LLM technology.

A review of the clinical spectrum of BRAT1 disorders and case of developmental and epileptic encephalopathy surviving into adulthood.

Pathogenic variants in BRAT1 are associated with a spectrum of clinical syndromes ranging from Lethal Neonatal Rigidity and Multifocal Seizure syndrome (RMFSL) to Neurodevelopmental Disorder with Cerebellar Atrophy and with or without Seizures (NEDCAS). RMFSL is characterized by early-onset multifocal seizures with microcephaly. Death occurs during infancy although a less severe course with later onset seizures and longer survival into childhood has been described. Here, we summarize published cases of BRAT1 disorders and present the case of a 20-year-old man with two heterozygous BRAT1 variants and a relatively later age of seizure onset with survival into adulthood. This case expands the spectrum of disease associated with BRAT1 variants and highlights the utility of genetic testing to identify the cause of developmental and epileptic encephalopathies where clinical heterogeneity within a spectrum of disease exists.

Late-onset cluster seizures and intellectual disability associated with a novel truncation variant in SMC1A.

SMC1A variants are known to cause Cornelia de Lange Syndrome (CdLS) which encompasses a clinical spectrum of intellectual disability, dysmorphic features (long or thick eyebrows, a hypomorphic philtrum and small nose) and, in some cases, epilepsy. More recently, SMC1A truncating variants have been described as the cause of a neurodevelopmental disorder with early-childhood onset drug-resistant epilepsy with seizures that occur in clusters, similar to that seen in PCDH19-related epilepsy, but without the classical features of CdLS. Here, we report the case of a 28-year-old woman with a de novo heterozygous truncating variant in SMC1A who unusually presented with seizures at the late age of 12 years and had normal development into adulthood.

Coronary artery disease in patients with dementia.

Our population is ageing. The prevalence of dementia is increasing as the population ages. Dementia is known to share many common risk factors with coronary artery disease including age, genetics, smoking, the components of the metabolic syndrome and inflammation. Despite the growing ageing population with dementia, there is underutilization of optimal care (pharmacotherapy and interventional procedures) in this cohort. Given common risk factors and potential benefit, patients with cognitive impairment and dementia should be offered contemporary care. However, further research evaluating optimal care in this patient cohort is warranted.