Performance evaluation of ChatGPT in detecting diagnostic errors and their contributing factors: an analysis of 545 case reports of diagnostic errors.

BACKGROUND: Manual chart review using validated assessment tools is a standardised methodology for detecting diagnostic errors. However, this requires considerable human resources and time. ChatGPT, a recently developed artificial intelligence chatbot based on a large language model, can effectively classify text based on suitable prompts. Therefore, ChatGPT can assist manual chart reviews in detecting diagnostic errors. OBJECTIVE: This study aimed to clarify whether ChatGPT could correctly detect diagnostic errors and possible factors contributing to them based on case presentations. METHODS: We analysed 545 published case reports that included diagnostic errors. We imputed the texts of case presentations and the final diagnoses with some original prompts into ChatGPT (GPT-4) to generate responses, including the judgement of diagnostic errors and contributing factors of diagnostic errors. Factors contributing to diagnostic errors were coded according to the following three taxonomies: Diagnosis Error Evaluation and Research (DEER), Reliable Diagnosis Challenges (RDC) and Generic Diagnostic Pitfalls (GDP). The responses on the contributing factors from ChatGPT were compared with those from physicians. RESULTS: ChatGPT correctly detected diagnostic errors in 519/545 cases (95%) and coded statistically larger numbers of factors contributing to diagnostic errors per case than physicians: DEER (median 5 vs 1, p<0.001), RDC (median 4 vs 2, p<0.001) and GDP (median 4 vs 1, p<0.001). The most important contributing factors of diagnostic errors coded by ChatGPT were 'failure/delay in considering the diagnosis' (315, 57.8%) in DEER, 'atypical presentation' (365, 67.0%) in RDC, and 'atypical presentation' (264, 48.4%) in GDP. CONCLUSION: ChatGPT accurately detects diagnostic errors from case presentations. ChatGPT may be more sensitive than manual reviewing in detecting factors contributing to diagnostic errors, especially for 'atypical presentation'.

Evaluating ChatGPT-4's Accuracy in Identifying Final Diagnoses Within Differential Diagnoses Compared With Those of Physicians: Experimental Study for Diagnostic Cases.

BACKGROUND: The potential of artificial intelligence (AI) chatbots, particularly ChatGPT with GPT-4 (OpenAI), in assisting with medical diagnosis is an emerging research area. However, it is not yet clear how well AI chatbots can evaluate whether the final diagnosis is included in differential diagnosis lists. OBJECTIVE: This study aims to assess the capability of GPT-4 in identifying the final diagnosis from differential-diagnosis lists and to compare its performance with that of physicians for case report series. METHODS: We used a database of differential-diagnosis lists from case reports in the American Journal of Case Reports, corresponding to final diagnoses. These lists were generated by 3 AI systems: GPT-4, Google Bard (currently Google Gemini), and Large Language Models by Meta AI 2 (LLaMA2). The primary outcome was focused on whether GPT-4's evaluations identified the final diagnosis within these lists. None of these AIs received additional medical training or reinforcement. For comparison, 2 independent physicians also evaluated the lists, with any inconsistencies resolved by another physician. RESULTS: The 3 AIs generated a total of 1176 differential diagnosis lists from 392 case descriptions. GPT-4's evaluations concurred with those of the physicians in 966 out of 1176 lists (82.1%). The Cohen κ coefficient was 0.63 (95% CI 0.56-0.69), indicating a fair to good agreement between GPT-4 and the physicians' evaluations. CONCLUSIONS: GPT-4 demonstrated a fair to good agreement in identifying the final diagnosis from differential-diagnosis lists, comparable to physicians for case report series. Its ability to compare differential diagnosis lists with final diagnoses suggests its potential to aid clinical decision-making support through diagnostic feedback. While GPT-4 showed a fair to good agreement for evaluation, its application in real-world scenarios and further validation in diverse clinical environments are essential to fully understand its utility in the diagnostic process.

Longitudinal Changes in Diagnostic Accuracy of a Differential Diagnosis List Developed by an AI-Based Symptom Checker: Retrospective Observational Study.

BACKGROUND: Artificial intelligence (AI) symptom checker models should be trained using real-world patient data to improve their diagnostic accuracy. Given that AI-based symptom checkers are currently used in clinical practice, their performance should improve over time. However, longitudinal evaluations of the diagnostic accuracy of these symptom checkers are limited. OBJECTIVE: This study aimed to assess the longitudinal changes in the accuracy of differential diagnosis lists created by an AI-based symptom checker used in the real world. METHODS: This was a single-center, retrospective, observational study. Patients who visited an outpatient clinic without an appointment between May 1, 2019, and April 30, 2022, and who were admitted to a community hospital in Japan within 30 days of their index visit were considered eligible. We only included patients who underwent an AI-based symptom checkup at the index visit, and the diagnosis was finally confirmed during follow-up. Final diagnoses were categorized as common or uncommon, and all cases were categorized as typical or atypical. The primary outcome measure was the accuracy of the differential diagnosis list created by the AI-based symptom checker, defined as the final diagnosis in a list of 10 differential diagnoses created by the symptom checker. To assess the change in the symptom checker's diagnostic accuracy over 3 years, we used a chi-square test to compare the primary outcome over 3 periods: from May 1, 2019, to April 30, 2020 (first year); from May 1, 2020, to April 30, 2021 (second year); and from May 1, 2021, to April 30, 2022 (third year). RESULTS: A total of 381 patients were included. Common diseases comprised 257 (67.5%) cases, and typical presentations were observed in 298 (78.2%) cases. Overall, the accuracy of the differential diagnosis list created by the AI-based symptom checker was 172 (45.1%), which did not differ across the 3 years (first year: 97/219, 44.3%; second year: 32/72, 44.4%; and third year: 43/90, 47.7%; P=.85). The accuracy of the differential diagnosis list created by the symptom checker was low in those with uncommon diseases (30/124, 24.2%) and atypical presentations (12/83, 14.5%). In the multivariate logistic regression model, common disease (P<.001; odds ratio 4.13, 95% CI 2.50-6.98) and typical presentation (P<.001; odds ratio 6.92, 95% CI 3.62-14.2) were significantly associated with the accuracy of the differential diagnosis list created by the symptom checker. CONCLUSIONS: A 3-year longitudinal survey of the diagnostic accuracy of differential diagnosis lists developed by an AI-based symptom checker, which has been implemented in real-world clinical practice settings, showed no improvement over time. Uncommon diseases and atypical presentations were independently associated with a lower diagnostic accuracy. In the future, symptom checkers should be trained to recognize uncommon conditions.

Genetic structure and diversity of the rfb locus of pathogenic species of the genus Leptospira.

Leptospirosis is caused by pathogenic strains of the genus Leptospira and is considered the most widespread zoonotic bacterial disease. The genus is characterized by the large number of serology variants, which challenges developing effective serotyping methods and vaccines with a broad spectrum. Because knowledge on the genetic basis of the serological diversity among leptospires is still limited, we aimed to explore the genetic structure and patterns of the rfb locus, which is involved in the biosynthesis of lipopolysaccharides, the major surface antigen that defines the serovar in leptospires. Here, we used genomic data of 722 pathogenic samples and compared the gene composition of their rfb locus by hierarchical clustering. Clustering analysis showed that the rfb locus gene composition is species-independent and strongly associated with the serological classification. The samples were grouped into four well-defined classes, which cluster together samples either belonging to the same serogroup or from different serogroups but sharing serological affinity. Our findings can assist in the development of new strategies based on molecular methods, which can lead to better tools for serological identification in this zoonosis.

Clinical decision support system using a machine learning model to assist simultaneous cardiopulmonary auscultation: Open-label randomized controlled trial.

Background: The utility of a clinical decision support system using a machine learning (ML) model for simultaneous cardiac and pulmonary auscultation is unknown. Objective: This study aimed to develop and evaluate an ML system's utility for cardiopulmonary auscultation. Methods: First, we developed an ML system for cardiopulmonary auscultation, using cardiopulmonary sound files from our previous study. The technique involved pre-processing, feature extraction, and classification through several neural network layers. After integration, the output class was categorized as "normal," "abnormal," or "undetermined." Second, we evaluated the ML system with 24 junior residents in an open-label randomized controlled trial at a university hospital. Participants were randomly assigned to the ML system group (intervention) or conventional auscultation group (control). During training, participants listened to four cardiac and four pulmonary sounds, all of which were correctly classified. Then, participants classified a series of 16 simultaneous cardiopulmonary sounds. The control group auscultated the sounds using noise-cancelling headphones, while the intervention group did so by watching recommendations from the ML system. Results: The total scores for correctly identified normal or abnormal cardiopulmonary sounds in the intervention group were significantly higher than those in the control group (366/384 [95.3%] vs. 343/384 [89.3%], P = 0.003). The cardiac test score in the intervention group was better (111/192 [57.8%] vs. 90/192 [46.9%], P = 0.04); there was no significant difference in pulmonary auscultation. Conclusions: The ML-based system improved the accuracy of cardiopulmonary auscultation for junior residents. This result suggests that the system can assist early-career physicians in accurate screening.

Effects of Combinational Use of Additional Differential Diagnostic Generators on the Diagnostic Accuracy of the Differential Diagnosis List Developed by an Artificial Intelligence-Driven Automated History-Taking System: Pilot Cross-Sectional Study.

BACKGROUND: Low diagnostic accuracy is a major concern in automated medical history-taking systems with differential diagnosis (DDx) generators. Extending the concept of collective intelligence to the field of DDx generators such that the accuracy of judgment becomes higher when accepting an integrated diagnosis list from multiple people than when accepting a diagnosis list from a single person may be a possible solution. OBJECTIVE: The purpose of this study is to assess whether the combined use of several DDx generators improves the diagnostic accuracy of DDx lists. METHODS: We used medical history data and the top 10 DDx lists (index DDx lists) generated by an artificial intelligence (AI)-driven automated medical history-taking system from 103 patients with confirmed diagnoses. Two research physicians independently created the other top 10 DDx lists (second and third DDx lists) per case by imputing key information into the other 2 DDx generators based on the medical history generated by the automated medical history-taking system without reading the index lists generated by the automated medical history-taking system. We used the McNemar test to assess the improvement in diagnostic accuracy from the index DDx lists to the three types of combined DDx lists: (1) simply combining DDx lists from the index, second, and third lists; (2) creating a new top 10 DDx list using a 1/n weighting rule; and (3) creating new lists with only shared diagnoses among DDx lists from the index, second, and third lists. We treated the data generated by 2 research physicians from the same patient as independent cases. Therefore, the number of cases included in analyses in the case using 2 additional lists was 206 (103 cases × 2 physicians' input). RESULTS: The diagnostic accuracy of the index lists was 46% (47/103). Diagnostic accuracy was improved by simply combining the other 2 DDx lists (133/206, 65%, P<.001), whereas the other 2 combined DDx lists did not improve the diagnostic accuracy of the DDx lists (106/206, 52%, P=.05 in the collective list with the 1/n weighting rule and 29/206, 14%, P<.001 in the only shared diagnoses among the 3 DDx lists). CONCLUSIONS: Simply adding each of the top 10 DDx lists from additional DDx generators increased the diagnostic accuracy of the DDx list by approximately 20%, suggesting that the combinational use of DDx generators early in the diagnostic process is beneficial.

Diagnostic errors in uncommon conditions: a systematic review of case reports of diagnostic errors.

OBJECTIVES: To assess the usefulness of case reports as sources for research on diagnostic errors in uncommon diseases and atypical presentations. CONTENT: We reviewed 563 case reports of diagnostic error. The commonality of the final diagnoses was classified based on the description in the articles, Orphanet, or epidemiological data on available references; the typicality of presentation was classified based on the description in the articles and the judgment of the physician researchers. Diagnosis Error Evaluation and Research (DEER), Reliable Diagnosis Challenges (RDC), and Generic Diagnostic Pitfalls (GDP) taxonomies were used to assess the factors contributing to diagnostic errors. SUMMARY AND OUTLOOK: Excluding three cases in that commonality could not be classified, 560 cases were classified into four categories: typical presentations of common diseases (60, 10.7 %), atypical presentations of common diseases (35, 6.2 %), typical presentations of uncommon diseases (276, 49.3 %), and atypical presentations of uncommon diseases (189, 33.8 %). The most important DEER taxonomy was "Failure/delay in considering the diagnosis" among the four categories, whereas the most important RDC and GDP taxonomies varied with the categories. Case reports can be a useful data source for research on the diagnostic errors of uncommon diseases with or without atypical presentations.

Effect of Prior Antibiotic Treatment on Blood Culture in an Outpatient Department of General Internal Medicine: A Retrospective Case-Control Analysis.

Purpose: The effect of antibiotics administered before blood cultures performed in general internal medicine outpatient settings is not well known. Patients and Methods: We conducted a retrospective case-control study including adult patients who underwent blood cultures in the general internal medicine outpatient department of a Japanese university hospital between 2016 and 2022. Patients with positive blood cultures were included as cases and matched patients with negative blood cultures were included as controls. Univariable and multivariable logistic regression analyses were performed. Results: A total of 200 patients and 200 controls were included. Antibiotics were administered prior to blood culture in 20% of patients (79/400). Oral antibiotics were prescribed to 69.6% of the prior antibiotics (55/79). Prior antibiotic use was significantly lower among patients with positive than negative blood cultures (13.5% vs 26.0%, p = 0.002) and was an independent predictive factor in univariable (odds ratio, 0.44; 95% confidence interval, 0.26-0.73; p = 0.002) and multivariable (adjusted odds ratio, 0.31; 95% confidence interval, 0.15-0.63; p = 0.002) logistic regression models for positive blood culture. The area under the receiver operating characteristic (AUROC) curve of the multivariable model for predicting positive blood cultures was 0.86. Conclusion: There was a negative correlation between prior antibiotic use and positive blood cultures in the general internal medicine outpatient department. Therefore, physicians should interpret the negative results of blood cultures performed after the administration of antibiotics with care.

Diagnostic Accuracy of Differential-Diagnosis Lists Generated by Generative Pretrained Transformer 3 Chatbot for Clinical Vignettes with Common Chief Complaints: A Pilot Study.

The diagnostic accuracy of differential diagnoses generated by artificial intelligence (AI) chatbots, including the generative pretrained transformer 3 (GPT-3) chatbot (ChatGPT-3) is unknown. This study evaluated the accuracy of differential-diagnosis lists generated by ChatGPT-3 for clinical vignettes with common chief complaints. General internal medicine physicians created clinical cases, correct diagnoses, and five differential diagnoses for ten common chief complaints. The rate of correct diagnosis by ChatGPT-3 within the ten differential-diagnosis lists was 28/30 (93.3%). The rate of correct diagnosis by physicians was still superior to that by ChatGPT-3 within the five differential-diagnosis lists (98.3% vs. 83.3%, p = 0.03). The rate of correct diagnosis by physicians was also superior to that by ChatGPT-3 in the top diagnosis (53.3% vs. 93.3%, p < 0.001). The rate of consistent differential diagnoses among physicians within the ten differential-diagnosis lists generated by ChatGPT-3 was 62/88 (70.5%). In summary, this study demonstrates the high diagnostic accuracy of differential-diagnosis lists generated by ChatGPT-3 for clinical cases with common chief complaints. This suggests that AI chatbots such as ChatGPT-3 can generate a well-differentiated diagnosis list for common chief complaints. However, the order of these lists can be improved in the future.

Natural genetic variation in the HAIRS ABSENT (H) gene increases type-VI glandular trichomes in both wild and domesticated tomatoes.

Glandular trichomes produce and exude secondary metabolites, conferring insect resistance in many crop species. Whereas some of its wild relatives are insect-resistant, tomato (Solanum lycopersicum) is not. Identifying the genetic changes that altered trichome development and biochemistry during tomato domestication would contribute to breeding for insect resistance. A mutation in the HAIRS ABSENT (H) gene, which encodes a C2H2 zinc finger protein (ZFP8), leads to reduced trichome density. Several geographic accessions of S. pimpinellifolium, the wild ancestor of domesticated tomato, have glabrous organs that resemble the phenotype caused by h. Here, we investigated allelic diversity for H in tomato and S. pimpinellifolium accessions and their associated trichome phenotypes. We also evaluated how the developmental stage can affect trichome development in glabrous and non-glabrous plants. We found that glabrous accessions of S. pimpinellifolium have different ZFP8 nucleotide sequence changes, associated with altered trichome development and density. We also found that while the glabrous appearance of h mutants is caused by a lower density of long trichomes, the density of type-VI glandular trichomes is increased, particularly in the adult stages of plant development. These insights on the genetic control of trichome development may contribute to breeding for insect resistance in tomatoes and other crops.

The systemic immune response due to cholesterol crystal embolization syndrome: a case report.

BACKGROUND: Cholesterol crystal embolization syndrome (CES) occurs when an atherosclerotic plaque causes small-vessel embolization, resulting in multi-organ damage. Although CES is pathologically characterized by an infiltration of eosinophils, the implication of the systemic inflammatory response represented by hypereosinophilia is unclear in clinical practice. Herein we present the case of a patient diagnosed with CES who developed multiple allergic organ injuries, including daptomycin-related dermatitis and later vancomycin-induced acute tubulointerstitial nephritis, which was successfully treated by the withdrawal of each medicine with or without corticosteroid therapy, one by one. CASE PRESENTATION: A 76-year-old Japanese man diagnosed with thoracic aneurysm rupture underwent total arch replacement through the open stent graft technique. Postoperatively, he developed methicillin-resistant Staphylococcus epidermidis bacteremia, which was treated with daptomycin. Subsequently, he presented with palpable purpura on both dorsal feet, erythema around his body, and hypereosinophilia. Daptomycin was replaced with vancomycin due to suspicion of drug-induced erythema. The erythema gradually faded. On nine days after vancomycin therapy, the systemic erythema rapidly reappeared followed by acute renal failure. The renal function decline prompted hemodialysis. A skin biopsy revealed cholesterol embolization, whereas a kidney biopsy revealed acute tubulointerstitial nephritis. After vancomycin discontinuation and initiation of systemic corticosteroid treatment, his kidney function was restored to the baseline level. CONCLUSIONS: The present case highlights cholesterol embolization can cause allergic complications in addition to direct organ damage.

Genetic basis underlying the serological affinity of leptospiral serovars from serogroups Sejroe, Mini and Hebdomadis.

Leptospirosis is a widely distributed zoonosis caused by pathogenic strains of bacteria of the genus Leptospira (Phylum Spirochaetes). Its agents are commonly classified based on their antigenic characteristics into serogroups and serovars, which are relevant for epidemiologic studies and vaccine development. Serological tests are considered laborious and require a specialized infrastructure. Some molecular methods have been proposed to accelerate these procedures, but they still can not replace the immunological tests, thus requiring a further understanding of the genetic basis underlying the serological classification. In this work, we focused on elucidating the genetic factors determinant for the serogroup Sejroe, which is one of the most prevalent serogroups in livestock. For this, we conducted a comparative analysis using >700 leptospiral genomic samples available in the public database. The analysis showed that the genes comprising the rfb locus are the main genetic factors associated with the serological classification. Samples from the serogroup Sejroe have an rfb locus with a conserved gene composition that differs from most other serogroups. Hebdomadis and Mini were the only serogroups whose samples have an rfb locus with similar gene composition to those from serogroup Sejroe, corroborating with the serological affinity shared by them. Finally, we could determine a small region in the rfb locus in which each of those three serogroups can be distinguished by its gene composition. This is the first work that uses an extensive repertoire of genomic data of leptospiral samples to elucidate the molecular basis of the serological classification and open the road to more reliable strategies based on molecular methods for serodiagnosis.

Auxin-driven ecophysiological diversification of leaves in domesticated tomato.

Heterobaric leaves have bundle sheath extensions (BSEs) that compartmentalize the parenchyma, whereas homobaric leaves do not. The presence of BSEs affects leaf hydraulics and photosynthetic rate. The tomato (Solanum lycopersicum) obscuravenosa (obv) mutant lacks BSEs. Here, we identify the obv gene and the causative mutation, a nonsynonymous amino acid change that disrupts a C2H2 zinc finger motif in a putative transcription factor. This mutation exists as a polymorphism in the natural range of wild tomatoes but has increased in frequency in domesticated tomatoes, suggesting that the latter diversified into heterobaric and homobaric leaf types. The obv mutant displays reduced vein density, leaf hydraulic conductance and photosynthetic assimilation rate. We show that these and other pleiotropic effects on plant development, including changes in leaf insertion angle, leaf margin serration, minor vein density, and fruit shape, are controlled by OBV via changes in auxin signaling. Loss of function of the transcriptional regulator AUXIN RESPONSE FACTOR 4 (ARF4) also results in defective BSE development, revealing an additional component of a genetic module controlling aspects of leaf development important for ecological adaptation and subject to breeding selection.

Taxallnomy: an extension of NCBI Taxonomy that produces a hierarchically complete taxonomic tree.

BACKGROUND: NCBI Taxonomy is the main taxonomic source for several bioinformatics tools and databases since all organisms with sequence accessions deposited on INSDC are organized in its hierarchical structure. Despite the extensive use and application of this data source, an alternative representation of data as a table would facilitate the use of information for processing bioinformatics data. To do so, since some taxonomic-ranks are missing in some lineages, an algorithm might propose provisional names for all taxonomic-ranks. RESULTS: To address this issue, we developed an algorithm that takes the tree structure from NCBI Taxonomy and generates a hierarchically complete taxonomic table, maintaining its compatibility with the original tree. The procedures performed by the algorithm consist of attempting to assign a taxonomic-rank to an existing clade or "no rank" node when possible, using its name as part of the created taxonomic-rank name (e.g. Ord_Ornithischia) or interpolating parent nodes when needed (e.g. Cla_of_Ornithischia), both examples given for the dinosaur Brachylophosaurus lineage. The new hierarchical structure was named Taxallnomy because it contains names for all taxonomic-ranks, and it contains 41 hierarchical levels corresponding to the 41 taxonomic-ranks currently found in the NCBI Taxonomy database. From Taxallnomy, users can obtain the complete taxonomic lineage with 41 nodes of all taxa available in the NCBI Taxonomy database, without any hazard to the original tree information. In this work, we demonstrate its applicability by embedding taxonomic information of a specified rank into a phylogenetic tree and by producing metagenomics profiles. CONCLUSION: Taxallnomy applies to any bioinformatics analyses that depend on the information from NCBI Taxonomy. Taxallnomy is updated periodically but with a distributed PERL script users can generate it locally using NCBI Taxonomy as input. All Taxallnomy resources are available at http://bioinfo.icb.ufmg.br/taxallnomy .

Plant Aquaporins: Diversity, Evolution and Biotechnological Applications.

The plasma membrane forms a permeable barrier that separates the cytoplasm from the external environment, defining the physical and chemical limits in each cell in all organisms. The movement of molecules and ions into and out of cells is controlled by the plasma membrane as a critical process for cell stability and survival, maintaining essential differences between the composition of the extracellular fluid and the cytosol. In this process aquaporins (AQPs) figure as important actors, comprising highly conserved membrane proteins that carry water, glycerol and other hydrophilic molecules through biomembranes, including the cell wall and membranes of cytoplasmic organelles. While mammals have 15 types of AQPs described so far (displaying 18 paralogs), a single plant species can present more than 120 isoforms, providing transport of different types of solutes. Such aquaporins may be present in the whole plant or can be associated with different tissues or situations, including biotic and especially abiotic stresses, such as drought, salinity or tolerance to soils rich in heavy metals, for instance. The present review addresses several aspects of plant aquaporins, from their structure, classification, and function, to in silico methodologies for their analysis and identification in transcriptomes and genomes. Aspects of evolution and diversification of AQPs (with a focus on plants) are approached for the first time with the aid of the LCA (Last Common Ancestor) analysis. Finally, the main practical applications involving the use of AQPs are discussed, including patents and future perspectives involving this important protein family.