RESUMEN
Introduction: Rheport is an online rheumatology referral system allowing automatic appointment triaging of new rheumatology patient referrals according to the respective probability of an inflammatory rheumatic disease (IRD). Previous research reported that Rheport was well accepted among IRD patients. Its accuracy was, however, limited, currently being based on an expert-based weighted sum score. This study aimed to evaluate whether machine learning (ML) models could improve this limited accuracy. Materials and methods: Data from a national rheumatology registry (RHADAR) was used to train and test nine different ML models to correctly classify IRD patients. Diagnostic performance was compared of ML models and the current algorithm was compared using the area under the receiver operating curve (AUROC). Feature importance was investigated using shapley additive explanation (SHAP). Results: A complete data set of 2265 patients was used to train and test ML models. 30.5% of patients were diagnosed with an IRD, 69.3% were female. The diagnostic accuracy of the current Rheport algorithm (AUROC of 0.534) could be improved with all ML models, (AUROC ranging between 0.630 and 0.737). Targeting a sensitivity of 90%, the logistic regression model could double current specificity (17% vs. 33%). Finger joint pain, inflammatory marker levels, psoriasis, symptom duration and female sex were the five most important features of the best performing logistic regression model for IRD classification. Conclusion: In summary, ML could improve the accuracy of a currently used rheumatology online referral system. Including further laboratory parameters and enabling individual feature importance adaption could increase accuracy and lead to broader usage.
RESUMEN
BACKGROUND: Timely diagnosis and treatment are essential in the effective management of inflammatory rheumatic diseases (IRDs). Symptom checkers (SCs) promise to accelerate diagnosis, reduce misdiagnoses, and guide patients more effectively through the health care system. Although SCs are increasingly used, there exists little supporting evidence. OBJECTIVE: To assess the diagnostic accuracy, patient-perceived usability, and acceptance of two SCs: (1) Ada and (2) Rheport. METHODS: Patients newly presenting to a German secondary rheumatology outpatient clinic were randomly assigned in a 1:1 ratio to complete Ada or Rheport and consecutively the respective other SCs in a prospective non-blinded controlled randomized crossover trial. The primary outcome was the accuracy of the SCs regarding the diagnosis of an IRD compared to the physicians' diagnosis as the gold standard. The secondary outcomes were patient-perceived usability, acceptance, and time to complete the SC. RESULTS: In this interim analysis, the first 164 patients who completed the study were analyzed. 32.9% (54/164) of the study subjects were diagnosed with an IRD. Rheport showed a sensitivity of 53.7% and a specificity of 51.8% for IRDs. Ada's top 1 (D1) and top 5 disease suggestions (D5) showed a sensitivity of 42.6% and 53.7% and a specificity of 63.6% and 54.5% concerning IRDs, respectively. The correct diagnosis of the IRD patients was within the Ada D1 and D5 suggestions in 16.7% (9/54) and 25.9% (14/54), respectively. The median System Usability Scale (SUS) score of Ada and Rheport was 75.0/100 and 77.5/100, respectively. The median completion time for both Ada and Rheport was 7.0 and 8.5 min, respectively. Sixty-four percent and 67.1% would recommend using Ada and Rheport to friends and other patients, respectively. CONCLUSIONS: While SCs are well accepted among patients, their diagnostic accuracy is limited to date. TRIAL REGISTRATION: DRKS.de, DRKS00017642 . Registered on 23 July 2019.
