Evaluating the accuracy and reliability of AI chatbots in disseminating the content of current resuscitation guidelines: a comparative analysis between the ERC 2021 guidelines and both ChatGPTs 3.5 and 4.

AIM OF THE STUDY: Artificial intelligence (AI) chatbots are established as tools for answering medical questions worldwide. Healthcare trainees are increasingly using this cutting-edge technology, although its reliability and accuracy in the context of healthcare remain uncertain. This study evaluated the suitability of Chat-GPT versions 3.5 and 4 for healthcare professionals seeking up-to-date evidence and recommendations for resuscitation by comparing the key messages of the resuscitation guidelines, which methodically set the gold standard of current evidence and recommendations, with the statements of the AI chatbots on this topic. METHODS:  This prospective comparative content analysis was conducted between the 2021 European Resuscitation Council (ERC) guidelines and the responses of two freely available ChatGPT versions (ChatGPT-3.5 and the Bing version of the ChatGPT-4) to questions about the key messages of clinically relevant ERC guideline chapters for adults. (1) The content analysis was performed bidirectionally by independent raters. The completeness and actuality of the AI output were assessed by comparing the key message with the AI-generated statements. (2) The conformity of the AI output was evaluated by comparing the statements of the two ChatGPT versions with the content of the ERC guidelines. RESULTS: In response to inquiries about the five chapters, ChatGPT-3.5 generated a total of 60 statements, whereas ChatGPT-4 produced 32 statements. ChatGPT-3.5 did not address 123 key messages, and ChatGPT-4 did not address 132 of the 172 key messages of the ERC guideline chapters. A total of 77% of the ChatGPT-3.5 statements and 84% of the ChatGPT-4 statements were fully in line with the ERC guidelines. The main reason for nonconformity was superficial and incorrect AI statements. The interrater reliability between the two raters, measured by Cohen's kappa, was greater for ChatGPT-4 (0.56 for completeness and 0.76 for conformity analysis) than for ChatGPT-3.5 (0.48 for completeness and 0.36 for conformity). CONCLUSION: We advise healthcare professionals not to rely solely on the tested AI-based chatbots to keep up to date with the latest evidence, as the relevant texts for the task were not part of the training texts of the underlying LLMs, and the lack of conceptual understanding of AI carries a high risk of spreading misconceptions. Original publications should always be considered for comprehensive understanding.

Risk prediction in stable cardiovascular disease using a high-sensitivity cardiac troponin T single biomarker strategy compared to the ESC-SCORE.

Objective: To evaluate the prognostic performance of high-sensitivity cardiac troponin T (hs-cTnT) compared with the ESC-SCORE. Methods: We included low-risk outpatients with stable cardiovascular (CV) disease categorised into need for non-secondary and secondary prevention. The prognostication of hs-cTnT at index visit was compared with the European Society of Cardiology-Systematic COronary Risk Evaluation (ESC-SCORE) with respect to all-cause mortality (ACM) and two composite endpoints (ACM, acute myocardial infarction (AMI) and stroke and ACM, AMI, stroke and rehospitalisation for acute coronary syndrome (ACS) and decompensated heart failure (DHF)). Results: Within a median follow-up of 796 days, a total of 16 deaths, 32 composite endpoints of ACM, AMI and stroke and 83 composite endpoints of ACM, AMI, stroke, rehospitalisation for ACS and DHF were observed among 693 stable low-risk outpatients. Using C-statistics, measurement of hs-cTnT alone outperformed the ESC-SCORE for the prediction of ACM in the entire study population (Δarea under the curve (AUC) 0.221, p=0.0039) and both prevention groups (non-secondary: ΔAUC 0.164, p=0.0208; secondary: ΔAUC 0.264, p=0.0134). For the prediction of all other secondary endpoints, hs-cTnT was at least as effective as the ESC-SCORE, both in secondary and non-secondary prevention. Using continuous and categorical net reclassification improvement and integrated discrimination improvement, hs-cTnT significantly improved reclassification regarding all endpoints in the entire population and in the secondary prevention cohort. In non-secondary prevention, hs-cTnT improved reclassification only for ACM. The results were confirmed in an independent external cohort on 2046 patients. Conclusions: Hs-cTnT is superior to the multivariable ESC-SCORE for the prediction of ACM and a composite endpoint in stable outpatients with and without relevant CV disease. Trial registration number: NCT01954303; Pre-results.

Prognostic Value of High-Sensitivity Cardiac Troponin T Compared with Risk Scores in Stable Cardiovascular Disease.

BACKGROUND: Risk stratification of patients with cardiovascular disease remains challenging despite consideration of risk scores. METHODS: We aimed to evaluate the prognostic performance of high-sensitivity cardiac troponin T in a low-risk outpatient population presenting for nonsecondary and secondary prevention. All-cause mortality, a composite of all-cause mortality, acute myocardial infarction, and stroke (end point 2), and a composite of all-cause mortality, acute myocardial infarction, stroke and rehospitalization for acute coronary syndrome, and decompensated heart failure (end point 3) were defined. The prognostic performance of high-sensitivity cardiac troponin T on index visit was compared with the PROCAM score and 3 FRAMINGHAM subscores. RESULTS: In 693 patients with a median follow-up of 796 days, we observed 16 deaths, 32 patients with end point 2, and 83 patients with end point 3. All risk scores performed better in the prediction of all-cause mortality in nonsecondary prevention (area under the curve [AUC]: PROCAM: 0.922 vs 0.523, P = .001, consistent for all other scores). In secondary prevention, high-sensitivity cardiac troponin T outperformed all risk scores in the prediction of all-cause mortality (ΔAUC: PROCAM: 0.319, P <.001, consistent for all other scores) and performed superiorly in the prediction of end point 2 compared with the PROCAM, FRAMINGHAM-Coronary Heart Disease, and FRAMINGHAM-Hard Coronary Heart Disease scores (ΔAUC: PROCAM: 0.176, P = .047, consistent for FRAMINGHAM-Coronary Heart Disease and FRAMINGHAM-Hard Coronary Heart Disease). In nonsecondary prevention, we observed a comparable prognostic performance of high-sensitivity cardiac troponin T and multivariable risk scores. Our findings on the prediction of all-cause mortality compared with the FRAMINGHAM-Hard Coronary Heart Disease score were confirmed in an independent validation cohort on 2046 patients. CONCLUSIONS: High-sensitivity troponin T provides excellent risk stratification regarding all-cause mortality and all-cause mortality, acute myocardial infarction, and stroke in a secondary prevention cohort in whom risk scores perform poorly.