Longitudinal Changes in Diagnostic Accuracy of a Differential Diagnosis List Developed by an AI-Based Symptom Checker: Retrospective Observational Study.

BACKGROUND: Artificial intelligence (AI) symptom checker models should be trained using real-world patient data to improve their diagnostic accuracy. Given that AI-based symptom checkers are currently used in clinical practice, their performance should improve over time. However, longitudinal evaluations of the diagnostic accuracy of these symptom checkers are limited. OBJECTIVE: This study aimed to assess the longitudinal changes in the accuracy of differential diagnosis lists created by an AI-based symptom checker used in the real world. METHODS: This was a single-center, retrospective, observational study. Patients who visited an outpatient clinic without an appointment between May 1, 2019, and April 30, 2022, and who were admitted to a community hospital in Japan within 30 days of their index visit were considered eligible. We only included patients who underwent an AI-based symptom checkup at the index visit, and the diagnosis was finally confirmed during follow-up. Final diagnoses were categorized as common or uncommon, and all cases were categorized as typical or atypical. The primary outcome measure was the accuracy of the differential diagnosis list created by the AI-based symptom checker, defined as the final diagnosis in a list of 10 differential diagnoses created by the symptom checker. To assess the change in the symptom checker's diagnostic accuracy over 3 years, we used a chi-square test to compare the primary outcome over 3 periods: from May 1, 2019, to April 30, 2020 (first year); from May 1, 2020, to April 30, 2021 (second year); and from May 1, 2021, to April 30, 2022 (third year). RESULTS: A total of 381 patients were included. Common diseases comprised 257 (67.5%) cases, and typical presentations were observed in 298 (78.2%) cases. Overall, the accuracy of the differential diagnosis list created by the AI-based symptom checker was 172 (45.1%), which did not differ across the 3 years (first year: 97/219, 44.3%; second year: 32/72, 44.4%; and third year: 43/90, 47.7%; P=.85). The accuracy of the differential diagnosis list created by the symptom checker was low in those with uncommon diseases (30/124, 24.2%) and atypical presentations (12/83, 14.5%). In the multivariate logistic regression model, common disease (P<.001; odds ratio 4.13, 95% CI 2.50-6.98) and typical presentation (P<.001; odds ratio 6.92, 95% CI 3.62-14.2) were significantly associated with the accuracy of the differential diagnosis list created by the symptom checker. CONCLUSIONS: A 3-year longitudinal survey of the diagnostic accuracy of differential diagnosis lists developed by an AI-based symptom checker, which has been implemented in real-world clinical practice settings, showed no improvement over time. Uncommon diseases and atypical presentations were independently associated with a lower diagnostic accuracy. In the future, symptom checkers should be trained to recognize uncommon conditions.

Evaluating ChatGPT-4's Diagnostic Accuracy: Impact of Visual Data Integration.

BACKGROUND: In the evolving field of health care, multimodal generative artificial intelligence (AI) systems, such as ChatGPT-4 with vision (ChatGPT-4V), represent a significant advancement, as they integrate visual data with text data. This integration has the potential to revolutionize clinical diagnostics by offering more comprehensive analysis capabilities. However, the impact on diagnostic accuracy of using image data to augment ChatGPT-4 remains unclear. OBJECTIVE: This study aims to assess the impact of adding image data on ChatGPT-4's diagnostic accuracy and provide insights into how image data integration can enhance the accuracy of multimodal AI in medical diagnostics. Specifically, this study endeavored to compare the diagnostic accuracy between ChatGPT-4V, which processed both text and image data, and its counterpart, ChatGPT-4, which only uses text data. METHODS: We identified a total of 557 case reports published in the American Journal of Case Reports from January 2022 to March 2023. After excluding cases that were nondiagnostic, pediatric, and lacking image data, we included 363 case descriptions with their final diagnoses and associated images. We compared the diagnostic accuracy of ChatGPT-4V and ChatGPT-4 without vision based on their ability to include the final diagnoses within differential diagnosis lists. Two independent physicians evaluated their accuracy, with a third resolving any discrepancies, ensuring a rigorous and objective analysis. RESULTS: The integration of image data into ChatGPT-4V did not significantly enhance diagnostic accuracy, showing that final diagnoses were included in the top 10 differential diagnosis lists at a rate of 85.1% (n=309), comparable to the rate of 87.9% (n=319) for the text-only version (P=.33). Notably, ChatGPT-4V's performance in correctly identifying the top diagnosis was inferior, at 44.4% (n=161), compared with 55.9% (n=203) for the text-only version (P=.002, χ2 test). Additionally, ChatGPT-4's self-reports showed that image data accounted for 30% of the weight in developing the differential diagnosis lists in more than half of cases. CONCLUSIONS: Our findings reveal that currently, ChatGPT-4V predominantly relies on textual data, limiting its ability to fully use the diagnostic potential of visual information. This study underscores the need for further development of multimodal generative AI systems to effectively integrate and use clinical image data. Enhancing the diagnostic performance of such AI systems through improved multimodal data integration could significantly benefit patient care by providing more accurate and comprehensive diagnostic insights. Future research should focus on overcoming these limitations, paving the way for the practical application of advanced AI in medicine.

Can ChatGPT-4 evaluate whether a differential diagnosis list contains the correct diagnosis as accurately as a physician?

OBJECTIVES: The potential of artificial intelligence (AI) chatbots, particularly the fourth-generation chat generative pretrained transformer (ChatGPT-4), in assisting with medical diagnosis is an emerging research area. While there has been significant emphasis on creating lists of differential diagnoses, it is not yet clear how well AI chatbots can evaluate whether the final diagnosis is included in these lists. This short communication aimed to assess the accuracy of ChatGPT-4 in evaluating lists of differential diagnosis compared to medical professionals' assessments. METHODS: We used ChatGPT-4 to evaluate whether the final diagnosis was included in the top 10 differential diagnosis lists created by physicians, ChatGPT-3, and ChatGPT-4, using clinical vignettes. Eighty-two clinical vignettes were used, comprising 52 complex case reports published by the authors from the department and 30 mock cases of common diseases created by physicians from the same department. We compared the agreement between ChatGPT-4 and the physicians on whether the final diagnosis was included in the top 10 differential diagnosis lists using the kappa coefficient. RESULTS: Three sets of differential diagnoses were evaluated for each of the 82 cases, resulting in a total of 246 lists. The agreement rate between ChatGPT-4 and physicians was 236 out of 246 (95.9 %), with a kappa coefficient of 0.86, indicating very good agreement. CONCLUSIONS: ChatGPT-4 demonstrated very good agreement with physicians in evaluating whether the final diagnosis should be included in the differential diagnosis lists.

Definitions and Measurements for Atypical Presentations at Risk for Diagnostic Errors in Internal Medicine: Protocol for a Scoping Review.

BACKGROUND: Atypical presentations have been increasingly recognized as a significant contributing factor to diagnostic errors in internal medicine. However, research to address associations between atypical presentations and diagnostic errors has not been evaluated due to the lack of widely applicable definitions and criteria for what is considered an atypical presentation. OBJECTIVE: The aim of the study is to describe how atypical presentations are defined and measured in studies of diagnostic errors in internal medicine and use this new information to develop new criteria to identify atypical presentations at high risk for diagnostic errors. METHODS: This study will follow an established framework for conducting scoping reviews. Inclusion criteria are developed according to the participants, concept, and context framework. This review will consider studies that fulfill all of the following criteria: include adult patients (participants); explore the association between atypical presentations and diagnostic errors using any definition, criteria, or measurement to identify atypical presentations and diagnostic errors (concept); and focus on internal medicine (context). Regarding the type of sources, this scoping review will consider quantitative, qualitative, and mixed methods study designs; systematic reviews; and opinion papers for inclusion. Case reports, case series, and conference abstracts will be excluded. The data will be extracted through MEDLINE, Web of Science, CINAHL, Embase, Cochrane Library, and Google Scholar searches. No limits will be applied to language, and papers indexed from database inception to December 31, 2023, will be included. Two independent reviewers (YH and RK) will conduct study selection and data extraction. The data extracted will include specific details about the patient characteristics (eg, age, sex, and disease), the definitions and measuring methods for atypical presentations and diagnostic errors, clinical settings (eg, department and outpatient or inpatient), type of evidence source, and the association between atypical presentations and diagnostic errors relevant to the review question. The extracted data will be presented in tabular format with descriptive statistics, allowing us to identify the key components or types of atypical presentations and develop new criteria to identify atypical presentations for future studies of diagnostic errors. Developing the new criteria will follow guidance for a basic qualitative content analysis with an inductive approach. RESULTS: As of January 2024, a literature search through multiple databases is ongoing. We will complete this study by December 2024. CONCLUSIONS: This scoping review aims to provide rigorous evidence to develop new criteria to identify atypical presentations at high risk for diagnostic errors in internal medicine. Such criteria could facilitate the development of a comprehensive conceptual model to understand the associations between atypical presentations and diagnostic errors in internal medicine. TRIAL REGISTRATION: Open Science Framework; www.osf.io/27d5m. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/56933.

Identifying Unmet Social Needs in a Patient Living in Isolation: A Case Report.

Loneliness and social isolation are common among older adults. To deliver high-quality care to older patients, healthcare professionals should know the social conditions of their patients. Addressing social determinants of health (SDH) in daily practice is beneficial to both patients and healthcare professionals. We illustrate a patient with congestive heart failure and cognitive decline whose social conditions improved through an SDH assessment. An SDH assessment has some potential advantages, which include facilitating a comprehensive understanding of patients' social conditions, visualizing how patients' social conditions have changed, deepening interprofessional collaboration, and ameliorating unnecessary negative emotions toward patients. This case report conveys two key messages. Firstly, healthcare professionals have the capability to evaluate patients' social backgrounds and enhance their health and social conditions through routine care. Secondly, the utilization of an SDH screening toolkit can support and enhance this initiative.

Clinical decision support system using a machine learning model to assist simultaneous cardiopulmonary auscultation: Open-label randomized controlled trial.

Background: The utility of a clinical decision support system using a machine learning (ML) model for simultaneous cardiac and pulmonary auscultation is unknown. Objective: This study aimed to develop and evaluate an ML system's utility for cardiopulmonary auscultation. Methods: First, we developed an ML system for cardiopulmonary auscultation, using cardiopulmonary sound files from our previous study. The technique involved pre-processing, feature extraction, and classification through several neural network layers. After integration, the output class was categorized as "normal," "abnormal," or "undetermined." Second, we evaluated the ML system with 24 junior residents in an open-label randomized controlled trial at a university hospital. Participants were randomly assigned to the ML system group (intervention) or conventional auscultation group (control). During training, participants listened to four cardiac and four pulmonary sounds, all of which were correctly classified. Then, participants classified a series of 16 simultaneous cardiopulmonary sounds. The control group auscultated the sounds using noise-cancelling headphones, while the intervention group did so by watching recommendations from the ML system. Results: The total scores for correctly identified normal or abnormal cardiopulmonary sounds in the intervention group were significantly higher than those in the control group (366/384 [95.3%] vs. 343/384 [89.3%], P = 0.003). The cardiac test score in the intervention group was better (111/192 [57.8%] vs. 90/192 [46.9%], P = 0.04); there was no significant difference in pulmonary auscultation. Conclusions: The ML-based system improved the accuracy of cardiopulmonary auscultation for junior residents. This result suggests that the system can assist early-career physicians in accurate screening.

Early swallowing rehabilitation and promotion of total oral intake in patients with aspiration pneumonia: A retrospective study.

OBJECTIVES: To investigate the impact of early swallowing assessment and rehabilitation on the total oral intake and in-hospital mortality in patients with aspiration pneumonia. METHODS: We retrospectively analyzed the data of patients with aspiration admitted between September 1, 2015, and October 31, 2016. The inclusion criterion was total oral intake before admission. A new protocol-based intervention for appropriate early oral intake was implemented on April 1, 2016. The protocol consisted of two steps. First, a screening test was conducted on the day of admission to detect patients who were not at high risk of dysphagia. Second, patients underwent a modified water swallowing test and water swallowing test. Patients cleared by these tests immediately initiated oral intake. The primary outcome, the composite outcomes of no recovery to total oral intake at discharge, and in-hospital mortality were compared between the patients admitted pre- and post protocol intervention. RESULTS: A total of 188 patients were included in the analysis (pre-, 92; post-, 96). The primary outcome did not differ between the pre- and post-intervention periods (23/92 [25.0%] vs. 18/96 [18.8%], p = 0.30). After adjusting for other variables, the intervention was significantly associated with a lower risk of composite outcomes (odds ratio, 0.22, 95%CI, 0.08-0.61, p = 0.004). CONCLUSION: The new protocol for early swallowing assessment, rehabilitation, and promotion of oral intake in patients admitted with aspiration pneumonia may be associated with the lower risk for the composite outcomes of in-hospital mortality and no recovery to total oral intake.

Three Subtypes of Cutaneous Nerve Entrapment Syndrome: A Narrative Review.

Anterior, lateral, and posterior cutaneous nerve entrapment syndromes have been proposed as etiologies of trunk pain. However, while these syndromes are analogous, comprehensive reports contrasting the three subtypes are lacking. We therefore reviewed the literature on anterior, lateral, and posterior cutaneous nerve entrapment syndrome. We searched the PubMed and Cochrane Library databases twice for relevant articles published between March and September 2022. In addition to 16 letters, technical reports, and review articles, a further 62, 6, and 3 articles concerning anterior, lateral, and posterior cutaneous nerve entrapment syndromes, respectively, were included. These syndromes are usually diagnosed based solely on unique history and examination findings; however, the diagnostic process may be prolonged, and multiple re-evaluations are required. The most common first-line treatment is trigger point injection; however, the management of refractory cases remains unclear. Awareness of this disease should be expanded to medical departments other than general medicine.

Prevalence of atypical presentations among outpatients and associations with diagnostic error.

OBJECTIVES: This study aimed to assess the prevalence of atypical presentations and their association with diagnostic errors in various diseases. METHODS: This retrospective observational study was conducted using cohort data between January 1 and December 31, 2019. Consecutive outpatients consulted by physicians from the Department of Diagnostic and Generalist Medicine at a university hospital in Japan were included. Patients for whom the final diagnosis was not confirmed were excluded. Primary outcomes were the prevalence of atypical presentations, and the prevalence of diagnostic errors in groups with typical and atypical presentations. Diagnostic errors and atypical presentations were assessed using the Revised Safer Dx Instrument. We performed primary analyses using a criterion; the average score of less than five to item 12 of two independent reviewers was an atypical presentation (liberal criterion). We also performed additional analyses using another criterion; the average score of three or less to item 12 was an atypical presentation (conservative criterion). RESULTS: A total of 930 patients were included out of a total of 2022 eligible. The prevalence of atypical presentation was 21.7 and 6.7 % when using liberal and conservative criteria for atypical presentation, respectively. Diagnostic errors (2.8 %) were most commonly observed in the cases with slight to moderate atypical presentation. Atypical presentation was associated with diagnostic errors with the liberal criterion for atypical presentation; however, this diminished with the conservative criterion. CONCLUSIONS: An atypical presentation was observed in up to 20 % of outpatients with a confirmed diagnosis, and slight to moderate atypical presentation may be the highest risk population for diagnostic errors.

Relationship Between Diabetic Chorea and Timing of MRI Findings: A Systematic Review with Case Reports.

Background: Diabetic chorea is a rare complication of diabetes mellitus for which head MRI is the most common diagnostic imaging modality. Cases have been reported where CT and/or MRI findings are inconsistent or clinical symptoms and imaging findings do not appear simultaneously. We aimed to compile the cases in which imaging findings appeared on MRI retests and to examine in a systematic review whether temporal differences in the appearance of imaging findings correlate with clinical characteristics. Case Presentation: An 80-year-old man with type 2 diabetes mellitus came to a hospital with abnormal movements of the left upper and lower extremities. Two days after the first visit, his symptoms flared up, and his head MRI showed an old cerebral infarction and no new lesion. On day 14, he retested T1-weighted imaging and showed a high signal in the right putamen, which was considered diabetic chorea. Blood glucose was controlled with insulin, and the involuntary movements disappeared. Methods: PubMed and ICHUSHI were searched to identify patients with diabetic chorea who had undergone MRI retests. Patients grouped by the temporal change in the presence/absence of imaging findings were compared on age, sex, duration of diabetes mellitus, blood glucose level, HbA1c level, side of involuntary movement, time to first MRI, and follow-up MRI. Results: Of the 64 cases analyzed, 43 (67.2%) were female. The mean age was 69.0 years. 16 (25.0%) had worsening findings upon MRI retesting, 37 (57.8%) had improvement, and 10 (15.6%) had unchanged findings. There were no significant differences in age, sex, mean blood glucose level or HbA1c at onset among the groups. Conclusion: There was no association between the pattern of appearance of imaging findings over time and clinical characteristics, including glucose levels. If initial MRI findings are negative, MRI retesting after a certain time may help diagnose diabetic chorea.

ChatGPT-Generated Differential Diagnosis Lists for Complex Case-Derived Clinical Vignettes: Diagnostic Accuracy Evaluation.

BACKGROUND: The diagnostic accuracy of differential diagnoses generated by artificial intelligence chatbots, including ChatGPT models, for complex clinical vignettes derived from general internal medicine (GIM) department case reports is unknown. OBJECTIVE: This study aims to evaluate the accuracy of the differential diagnosis lists generated by both third-generation ChatGPT (ChatGPT-3.5) and fourth-generation ChatGPT (ChatGPT-4) by using case vignettes from case reports published by the Department of GIM of Dokkyo Medical University Hospital, Japan. METHODS: We searched PubMed for case reports. Upon identification, physicians selected diagnostic cases, determined the final diagnosis, and displayed them into clinical vignettes. Physicians typed the determined text with the clinical vignettes in the ChatGPT-3.5 and ChatGPT-4 prompts to generate the top 10 differential diagnoses. The ChatGPT models were not specially trained or further reinforced for this task. Three GIM physicians from other medical institutions created differential diagnosis lists by reading the same clinical vignettes. We measured the rate of correct diagnosis within the top 10 differential diagnosis lists, top 5 differential diagnosis lists, and the top diagnosis. RESULTS: In total, 52 case reports were analyzed. The rates of correct diagnosis by ChatGPT-4 within the top 10 differential diagnosis lists, top 5 differential diagnosis lists, and top diagnosis were 83% (43/52), 81% (42/52), and 60% (31/52), respectively. The rates of correct diagnosis by ChatGPT-3.5 within the top 10 differential diagnosis lists, top 5 differential diagnosis lists, and top diagnosis were 73% (38/52), 65% (34/52), and 42% (22/52), respectively. The rates of correct diagnosis by ChatGPT-4 were comparable to those by physicians within the top 10 (43/52, 83% vs 39/52, 75%, respectively; P=.47) and within the top 5 (42/52, 81% vs 35/52, 67%, respectively; P=.18) differential diagnosis lists and top diagnosis (31/52, 60% vs 26/52, 50%, respectively; P=.43) although the difference was not significant. The ChatGPT models' diagnostic accuracy did not significantly vary based on open access status or the publication date (before 2011 vs 2022). CONCLUSIONS: This study demonstrates the potential diagnostic accuracy of differential diagnosis lists generated using ChatGPT-3.5 and ChatGPT-4 for complex clinical vignettes from case reports published by the GIM department. The rate of correct diagnoses within the top 10 and top 5 differential diagnosis lists generated by ChatGPT-4 exceeds 80%. Although derived from a limited data set of case reports from a single department, our findings highlight the potential utility of ChatGPT-4 as a supplementary tool for physicians, particularly for those affiliated with the GIM department. Further investigations should explore the diagnostic accuracy of ChatGPT by using distinct case materials beyond its training data. Such efforts will provide a comprehensive insight into the role of artificial intelligence in enhancing clinical decision-making.

Effects of Combinational Use of Additional Differential Diagnostic Generators on the Diagnostic Accuracy of the Differential Diagnosis List Developed by an Artificial Intelligence-Driven Automated History-Taking System: Pilot Cross-Sectional Study.

BACKGROUND: Low diagnostic accuracy is a major concern in automated medical history-taking systems with differential diagnosis (DDx) generators. Extending the concept of collective intelligence to the field of DDx generators such that the accuracy of judgment becomes higher when accepting an integrated diagnosis list from multiple people than when accepting a diagnosis list from a single person may be a possible solution. OBJECTIVE: The purpose of this study is to assess whether the combined use of several DDx generators improves the diagnostic accuracy of DDx lists. METHODS: We used medical history data and the top 10 DDx lists (index DDx lists) generated by an artificial intelligence (AI)-driven automated medical history-taking system from 103 patients with confirmed diagnoses. Two research physicians independently created the other top 10 DDx lists (second and third DDx lists) per case by imputing key information into the other 2 DDx generators based on the medical history generated by the automated medical history-taking system without reading the index lists generated by the automated medical history-taking system. We used the McNemar test to assess the improvement in diagnostic accuracy from the index DDx lists to the three types of combined DDx lists: (1) simply combining DDx lists from the index, second, and third lists; (2) creating a new top 10 DDx list using a 1/n weighting rule; and (3) creating new lists with only shared diagnoses among DDx lists from the index, second, and third lists. We treated the data generated by 2 research physicians from the same patient as independent cases. Therefore, the number of cases included in analyses in the case using 2 additional lists was 206 (103 cases × 2 physicians' input). RESULTS: The diagnostic accuracy of the index lists was 46% (47/103). Diagnostic accuracy was improved by simply combining the other 2 DDx lists (133/206, 65%, P<.001), whereas the other 2 combined DDx lists did not improve the diagnostic accuracy of the DDx lists (106/206, 52%, P=.05 in the collective list with the 1/n weighting rule and 29/206, 14%, P<.001 in the only shared diagnoses among the 3 DDx lists). CONCLUSIONS: Simply adding each of the top 10 DDx lists from additional DDx generators increased the diagnostic accuracy of the DDx list by approximately 20%, suggesting that the combinational use of DDx generators early in the diagnostic process is beneficial.

Diagnostic errors in uncommon conditions: a systematic review of case reports of diagnostic errors.

OBJECTIVES: To assess the usefulness of case reports as sources for research on diagnostic errors in uncommon diseases and atypical presentations. CONTENT: We reviewed 563 case reports of diagnostic error. The commonality of the final diagnoses was classified based on the description in the articles, Orphanet, or epidemiological data on available references; the typicality of presentation was classified based on the description in the articles and the judgment of the physician researchers. Diagnosis Error Evaluation and Research (DEER), Reliable Diagnosis Challenges (RDC), and Generic Diagnostic Pitfalls (GDP) taxonomies were used to assess the factors contributing to diagnostic errors. SUMMARY AND OUTLOOK: Excluding three cases in that commonality could not be classified, 560 cases were classified into four categories: typical presentations of common diseases (60, 10.7 %), atypical presentations of common diseases (35, 6.2 %), typical presentations of uncommon diseases (276, 49.3 %), and atypical presentations of uncommon diseases (189, 33.8 %). The most important DEER taxonomy was "Failure/delay in considering the diagnosis" among the four categories, whereas the most important RDC and GDP taxonomies varied with the categories. Case reports can be a useful data source for research on the diagnostic errors of uncommon diseases with or without atypical presentations.