Diagnostic Accuracy of a Mobile AI-Based Symptom Checker and a Web-Based Self-Referral Tool in Rheumatology: Multicenter Randomized Controlled Trial.

BACKGROUND: The diagnosis of inflammatory rheumatic diseases (IRDs) is often delayed due to unspecific symptoms and a shortage of rheumatologists. Digital diagnostic decision support systems (DDSSs) have the potential to expedite diagnosis and help patients navigate the health care system more efficiently. OBJECTIVE: The aim of this study was to assess the diagnostic accuracy of a mobile artificial intelligence (AI)-based symptom checker (Ada) and a web-based self-referral tool (Rheport) regarding IRDs. METHODS: A prospective, multicenter, open-label, crossover randomized controlled trial was conducted with patients newly presenting to 3 rheumatology centers. Participants were randomly assigned to complete a symptom assessment using either Ada or Rheport. The primary outcome was the correct identification of IRDs by the DDSSs, defined as the presence of any IRD in the list of suggested diagnoses by Ada or achieving a prespecified threshold score with Rheport. The gold standard was the diagnosis made by rheumatologists. RESULTS: A total of 600 patients were included, among whom 214 (35.7%) were diagnosed with an IRD. Most frequent IRD was rheumatoid arthritis with 69 (11.5%) patients. Rheport's disease suggestion and Ada's top 1 (D1) and top 5 (D5) disease suggestions demonstrated overall diagnostic accuracies of 52%, 63%, and 58%, respectively, for IRDs. Rheport showed a sensitivity of 62% and a specificity of 47% for IRDs. Ada's D1 and D5 disease suggestions showed a sensitivity of 52% and 66%, respectively, and a specificity of 68% and 54%, respectively, concerning IRDs. Ada's diagnostic accuracy regarding individual diagnoses was heterogenous, and Ada performed considerably better in identifying rheumatoid arthritis in comparison to other diagnoses (D1: 42%; D5: 64%). The Cohen κ statistic of Rheport for agreement on any rheumatic disease diagnosis with Ada D1 was 0.15 (95% CI 0.08-0.18) and with Ada D5 was 0.08 (95% CI 0.00-0.16), indicating poor agreement for the presence of any rheumatic disease between the 2 DDSSs. CONCLUSIONS: To our knowledge, this is the largest comparative DDSS trial with actual use of DDSSs by patients. The diagnostic accuracies of both DDSSs for IRDs were not promising in this high-prevalence patient population. DDSSs may lead to a misuse of scarce health care resources. Our results underscore the need for stringent regulation and drastic improvements to ensure the safety and efficacy of DDSSs. TRIAL REGISTRATION: German Register of Clinical Trials DRKS00017642; https://drks.de/search/en/trial/DRKS00017642.

Comparison of physician and artificial intelligence-based symptom checker diagnostic accuracy.

Symptom checkers are increasingly used to assess new symptoms and navigate the health care system. The aim of this study was to compare the accuracy of an artificial intelligence (AI)-based symptom checker (Ada) and physicians regarding the presence/absence of an inflammatory rheumatic disease (IRD). In this survey study, German-speaking physicians with prior rheumatology working experience were asked to determine IRD presence/absence and suggest diagnoses for 20 different real-world patient vignettes, which included only basic health and symptom-related medical history. IRD detection rate and suggested diagnoses of participants and Ada were compared to the gold standard, the final rheumatologists' diagnosis, reported on the discharge summary report. A total of 132 vignettes were completed by 33 physicians (mean rheumatology working experience 8.8 (SD 7.1) years). Ada's diagnostic accuracy (IRD) was significantly higher compared to physicians (70 vs 54%, p = 0.002) according to top diagnosis. Ada listed the correct diagnosis more often compared to physicians (54 vs 32%, p < 0.001) as top diagnosis as well as among the top 3 diagnoses (59 vs 42%, p < 0.001). Work experience was not related to suggesting the correct diagnosis or IRD status. Confined to basic health and symptom-related medical history, the diagnostic accuracy of physicians was lower compared to an AI-based symptom checker. These results highlight the potential of using symptom checkers early during the patient journey and importance of access to complete and sufficient patient information to establish a correct diagnosis.

Application of visually based, computerised diagnostic decision support system in dermatological medical education: a pilot study.

BACKGROUND: Medical education has shifted from memory-based practice to evidence-based decisions. The question arises: how can we ensure that all students get correct and systematic information? Visually based, computerised diagnostic decision support system (VCDDSS, VisualDx) may just fit our needs. A pilot study was conducted to investigate its role in medical education and clinical practice. METHODS: This was a prospective study, including one consultant dermatologist, 51 medical students and 13 dermatology residents, conducted in the dermatology teaching clinic at China Medical University Hospital from 30 December 2014 to 21 April 2015. Clinical diagnoses of 13 patients were made before and after using VCDDSS. Questionnaires were filled out at the end. The consultant dermatologist's diagnosis was defined as the standard answer; the Sign test was used to analyse diagnostic accuracy and the Fisher exact test to analyse questionnaires. RESULTS: There was an 18.75% increase in diagnostic accuracy after use of VCDDSS (62.5-81.25%; p value <0.01). Significant associations were found in diagnostic assistance in terms of user factors such as accessibility, interface satisfaction, quality of imaging, textual description, and a Chinese language interface option (p value<0.01). CONCLUSIONS: This study demonstrated that VCDDSS increases diagnostic accuracy by 18.75%, which means we can avoid possible misdiagnosis, provide better treatment, and avoid waste of medical resources. The user satisfaction is high. We expect wider application of this kind of decision support system in clinical practice, medical education, residency training, and patient education in the future. Further large-scale studies should be planned to confirm its application.

A Novel Diagnostic Decision Support System for Medical Professionals: Prospective Feasibility Study.

BACKGROUND: Continuously growing medical knowledge and the increasing amount of data make it difficult for medical professionals to keep track of all new information and to place it in the context of existing information. A variety of digital technologies and artificial intelligence-based methods are currently available as persuasive tools to empower physicians in clinical decision-making and improve health care quality. A novel diagnostic decision support system (DDSS) prototype developed by Ada Health GmbH with a focus on traceability, transparency, and usability will be examined more closely in this study. OBJECTIVE: The aim of this study is to test the feasibility and functionality of a novel DDSS prototype, exploring its potential and performance in identifying the underlying cause of acute dyspnea in patients at the University Hospital Basel. METHODS: A prospective, observational feasibility study was conducted at the emergency department (ED) and internal medicine ward of the University Hospital Basel, Switzerland. A convenience sample of 20 adult patients admitted to the ED with dyspnea as the chief complaint and a high probability of inpatient admission was selected. A study physician followed the patients admitted to the ED throughout the hospitalization without interfering with the routine clinical work. Routinely collected health-related personal data from these patients were entered into the DDSS prototype. The DDSS prototype's resulting disease probability list was compared with the gold-standard main diagnosis provided by the treating physician. RESULTS: The DDSS presented information with high clarity and had a user-friendly, novel, and transparent interface. The DDSS prototype was not perfectly suited for the ED as case entry was time-consuming (1.5-2 hours per case). It provided accurate decision support in the clinical inpatient setting (average of cases in which the correct diagnosis was the first diagnosis listed: 6/20, 30%, SD 2.10%; average of cases in which the correct diagnosis was listed as one of the top 3: 11/20, 55%, SD 2.39%; average of cases in which the correct diagnosis was listed as one of the top 5: 14/20, 70%, SD 2.26%) in patients with dyspnea as the main presenting complaint. CONCLUSIONS: The study of the feasibility and functionality of the tool was successful, with some limitations. Used in the right place, the DDSS has the potential to support physicians in their decision-making process by showing new pathways and unintentionally ignored diagnoses. The DDSS prototype had some limitations regarding the process of data input, diagnostic accuracy, and completeness of the integrated medical knowledge. The results of this study provide a basis for the tool's further development. In addition, future studies should be conducted with the aim to overcome the current limitations of the tool and study design. TRIAL REGISTRATION: ClinicalTrials.gov NCT04827342; https://clinicaltrials.gov/ct2/show/NCT04827342.

Patient's Perception of Digital Symptom Assessment Technologies in Rheumatology: Results From a Multicentre Study.

Introduction: An increasing number of digital tools, including dedicated diagnostic decision support systems (DDSS) exist to better assess new symptoms and understand when and where to seek medical care. The aim of this study was to evaluate patient's previous online assessment experiences and to compare the acceptability, usability, usefulness and potential impact of artificial intelligence (AI)-based symptom checker (Ada) and an online questionnaire-based self-referral tool (Rheport). Materials and Methods: Patients newly presenting to three German secondary rheumatology outpatient clinics were randomly assigned in a 1:1 ratio to complete consecutively Ada or Rheport in a prospective non-blinded multicentre controlled crossover randomized trial. DDSS completion time was recorded by local study personnel and perceptions on DDSS and previous online assessment were collected through a self-completed study questionnaire, including usability measured with the validated System Usability Scale (SUS). Results: 600 patients (median age 52 years, 418 women) were included. 277/600 (46.2%) of patients used an online search engine prior to the appointment. The median time patients spent assessing symptoms was 180, 7, and 8 min, respectively using online using search engines, Ada and Rheport. 111/275 (40.4%), 266/600 (44.3%) and 395/600 (65.8%) of patients rated the respective symptom assessment as very helpful or helpful, using online search engines, Ada and Rheport, respectively. Usability of both diagnostic decision support systems (DDSS) was "good" with a significantly higher mean SUS score (SD) of Rheport 77.1/100 (16.0) compared to Ada 74.4/100 (16.8), (p < 0.0001). In male patients, usability of Rheport was rated higher than Ada (p = 0.02) and the usability rating of older (52 years ≥) patients of both DDSS was lower than in younger participants (p = 0.005). Both effects were independent of each other. 440/600 (73.3%) and 475/600 (79.2%) of the patients would recommend Ada and Rheport to friends and other patients, respectively. Conclusion: In summary, patients increasingly assess their symptoms independently online, however only a minority used dedicated symptom assessment websites or DDSS. DDSS, such as Ada an Rheport are easy to use, well accepted among patients with musculoskeletal complaints and could replace online search engines for patient symptom assessment, potentially saving time and increasing helpfulness.

A Novel Bayesian General Medical Diagnostic Assistant Achieves Superior Accuracy With Sparse History: A Performance Comparison of 7 Online Diagnostic Aids and Physicians.

Online AI symptom checkers and diagnostic assistants (DAs) have tremendous potential to reduce misdiagnosis and cost, while increasing the quality, convenience, and availability of healthcare, but only if they can perform with high accuracy. We introduce a novel Bayesian DA designed to improve diagnostic accuracy by addressing key weaknesses of Bayesian Network implementations for clinical diagnosis. We compare the performance of our prototype DA (MidasMed) to that of physicians and six other publicly accessible DAs (Ada, Babylon, Buoy, Isabel, Symptomate, and WebMD) using a set of 30 publicly available case vignettes, and using only sparse history (no exam findings or tests). Our results demonstrate superior performance of the MidasMed DA, with the correct diagnosis being the top ranked disorder in 93% of cases, and in the top 3 in 96% of cases.

Development of a Machine Learning Classifier Based on Radiomic Features Extracted From Post-Contrast 3D T1-Weighted MR Images to Distinguish Glioblastoma From Solitary Brain Metastasis.

OBJECTIVES: To differentiate Glioblastomas (GBM) and Brain Metastases (BM) using a radiomic features-based Machine Learning (ML) classifier trained from post-contrast three-dimensional T1-weighted (post-contrast 3DT1) MR imaging, and compare its performance in medical diagnosis versus human experts, on a testing cohort. METHODS: We enrolled 143 patients (71 GBM and 72 BM) in a retrospective bicentric study from January 2010 to May 2019 to train the classifier. Post-contrast 3DT1 MR images were performed on a 3-Tesla MR unit and 100 radiomic features were extracted. Selection and optimization of the Machine Learning (ML) classifier was performed using a nested cross-validation. Sensitivity, specificity, balanced accuracy, and area under the receiver operating characteristic curve (AUC) were calculated as performance metrics. The model final performance was cross-validated, then evaluated on a test set of 37 patients, and compared to human blind reading using a McNemar's test. RESULTS: The ML classifier had a mean [95% confidence interval] sensitivity of 85% [77; 94], a specificity of 87% [78; 97], a balanced accuracy of 86% [80; 92], and an AUC of 92% [87; 97] with cross-validation. Sensitivity, specificity, balanced accuracy and AUC were equal to 75, 86, 80 and 85% on the test set. Sphericity 3D radiomic index highlighted the highest coefficient in the logistic regression model. There were no statistical significant differences observed between the performance of the classifier and the experts' blinded examination. CONCLUSIONS: The proposed diagnostic support system based on radiomic features extracted from post-contrast 3DT1 MR images helps in differentiating solitary BM from GBM with high diagnosis performance and generalizability.

Clinician-centric diagnosis of rare genetic diseases: performance of a gene pertinence metric in decision support for clinicians.

BACKGROUND: In diagnosis of rare genetic diseases we face a decision as to the degree to which the sequencing lab offers one or more diagnoses based on clinical input provided by the clinician, or the clinician reaches a diagnosis based on the complete set of variants provided by the lab. We tested a software approach to assist the clinician in making the diagnosis based on clinical findings and an annotated genomic variant table, using cases already solved using less automated processes. RESULTS: For the 81 cases studied (involving 216 individuals), 70 had genetic abnormalities with phenotypes previously described in the literature, and 11 were not described in the literature at the time of analysis ("discovery genes"). These included cases beyond a trio, including ones with different variants in the same gene. In 100% of cases the abnormality was recognized. Of the 70, the abnormality was ranked #1 in 94% of cases, with an average rank 1.1 for all cases. Large CNVs could be analyzed in an integrated analysis, performed in 24 of the cases. The process is rapid enough to allow for periodic reanalysis of unsolved cases. CONCLUSIONS: A clinician-friendly environment for clinical correlation can be provided to clinicians who are best positioned to have the clinical information needed for this interpretation.

Can a decision support system accelerate rare disease diagnosis? Evaluating the potential impact of Ada DX in a retrospective study.

BACKGROUND: Rare disease diagnosis is often delayed by years. A primary factor for this delay is a lack of knowledge and awareness regarding rare diseases. Probabilistic diagnostic decision support systems (DDSSs) have the potential to accelerate rare disease diagnosis by suggesting differential diagnoses for physicians based on case input and incorporated medical knowledge. We examine the DDSS prototype Ada DX and assess its potential to provide accurate rare disease suggestions early in the course of rare disease cases. RESULTS: Ada DX suggested the correct disease earlier than the time of clinical diagnosis among the top five fit disease suggestions in 53.8% of cases (50 of 93), and as the top fit disease suggestion in 37.6% of cases (35 of 93). The median advantage of correct disease suggestions compared to the time of clinical diagnosis was 3 months or 50% for top five fit and 1 month or 21% for top fit. The correct diagnosis was suggested at the first documented patient visit in 33.3% (top 5 fit), and 16.1% of cases (top fit), respectively. Wilcoxon signed-rank test shows a significant difference between the time to clinical diagnosis and the time to correct disease suggestion for both top five fit and top fit (z-score -6.68, respective -5.71, α=0.05, p-value <0.001). CONCLUSION: Ada DX provided accurate rare disease suggestions in most rare disease cases. In many cases, Ada DX provided correct rare disease suggestions early in the course of the disease, sometimes at the very beginning of a patient journey. The interpretation of these results indicates that Ada DX has the potential to suggest rare diseases to physicians early in the course of a case. Limitations of this study derive from its retrospective and unblinded design, data input by a single user, and the optimization of the knowledge base during the course of the study. Results pertaining to the system's accuracy should be interpreted cautiously. Whether the use of Ada DX reduces the time to diagnosis in rare diseases in a clinical setting should be validated in prospective studies.