Development and Clinical Evaluation of an Artificial Intelligence Support Tool for Improving Telemedicine Photo Quality.

Importance: Telemedicine use accelerated during the COVID-19 pandemic, and skin conditions were a common use case. However, many images submitted may be of insufficient quality for making a clinical determination. Objective: To determine whether an artificial intelligence (AI) decision support tool, a machine learning algorithm, could improve the quality of images submitted for telemedicine by providing real-time feedback and explanations to patients. Design, Setting, and Participants: This quality improvement study with an AI performance component and single-arm clinical pilot study component was conducted from March 2020 to October 2021. After training, the AI decision support tool was tested on 357 retrospectively collected telemedicine images from Stanford telemedicine from March 2020 to June 2021. Subsequently, a single-arm clinical pilot study was conducted to assess feasibility with 98 patients in the Stanford Department of Dermatology across 2 clinical sites from July 2021 to October 2021. For the clinical pilot study, inclusion criteria for patients included being adults (aged ≥18 years), presenting to clinic for a skin condition, and being able to photograph their own skin with a smartphone. Interventions: During the clinical pilot study, patients were given a handheld smartphone device with a machine learning algorithm interface loaded and were asked to take images of any lesions of concern. Patients were able to review and retake photos prior to submitting, so each submitted photo met the patient's assumed standard of clinical acceptability. A machine learning algorithm then gave the patient feedback on whether the image was acceptable. If the image was rejected, the patient was provided a reason by the AI decision support tool and allowed to retake the photos. Main Outcomes and Measures: The main outcome of the retrospective image analysis was the receiver operator curve area under the curve (ROC-AUC). The main outcome of the clinical pilot study was the image quality difference between the baseline images and the images approved by AI decision support. Results: Of the 98 patients included, the mean (SD) age was 49.8 (17.6) years, and 50 (51%) of the patients were male. On retrospective telemedicine images, the machine learning algorithm effectively identified poor-quality images (ROC-AUC of 0.78) and the reason for poor quality (blurry ROC-AUC of 0.84; lighting issues ROC-AUC of 0.70). The performance was consistent across age and sex. In the clinical pilot study, patient use of the machine learning algorithm was associated with improved image quality. An AI algorithm was associated with reduction in the number of patients with a poor-quality image by 68.0%. Conclusions and Relevance: In this quality improvement study, patients use of the AI decision support with a machine learning algorithm was associated with improved quality of skin disease photographs submitted for telemedicine use.

Disparities in dermatology AI performance on a diverse, curated clinical image set.

An estimated 3 billion people lack access to dermatological care globally. Artificial intelligence (AI) may aid in triaging skin diseases and identifying malignancies. However, most AI models have not been assessed on images of diverse skin tones or uncommon diseases. Thus, we created the Diverse Dermatology Images (DDI) dataset-the first publicly available, expertly curated, and pathologically confirmed image dataset with diverse skin tones. We show that state-of-the-art dermatology AI models exhibit substantial limitations on the DDI dataset, particularly on dark skin tones and uncommon diseases. We find that dermatologists, who often label AI datasets, also perform worse on images of dark skin tones and uncommon diseases. Fine-tuning AI models on the DDI images closes the performance gap between light and dark skin tones. These findings identify important weaknesses and biases in dermatology AI that should be addressed for reliable application to diverse patients and diseases.

Predicting Visuo-Motor Diseases From Eye Tracking Data.

Eye tracking, or oculography, provides insight into where a person is looking. Recent advances in camera technology and machine learning have enabled prevalent devices like smart-phones to track gaze and visuo-motor behavior at near clinical-quality resolution. A critical gap in using oculography to diagnose visuo-motor dysfunction on a large scale is in the design of visual task paradigms, algorithms for diagnosis, and sufficiently large datasets. In this study, we used a 500 Hz infrared oculography dataset in healthy controls and patients with various neurological diseases causing visuo-motor abnormality due to eye movement disorder or vision loss. We used novel visuo-motor tasks involving rapid reading of 40 single-digit numbers per page and developed a machine learning algorithm for predicting disease state. We show that oculography data acquired while a person reads one page of 40 single-digit numbers (15-30 seconds duration) is predictive of of visuo-motor dysfunction (ROC-AUC = 0:973). Remarkably, we also find that short recordings of about 2.5 seconds (6-12× reduction in time) are sufficient for disease detection (ROC-AUC = 0:831). We identify which tasks are most informative for identifying visuo-motor dysfunction (those with the most visual crowding), and more specifically, which aspects of the task are most predictive (the recording segments where gaze moves vertically across lines). In addition to segregating disease and controls, our novel visuo-motor paradigms can discriminate among diseases impacting eye movement, diseases associated with vision loss, and healthy controls (81% accuracy compared with baseline of 33%).

TrueImage: A Machine Learning Algorithm to Improve the Quality of Telehealth Photos.

Telehealth is an increasingly critical component of the health care ecosystem, especially due to the COVID-19 pandemic. Rapid adoption of telehealth has exposed limitations in the existing infrastructure. In this paper, we study and highlight photo quality as a major challenge in the telehealth workflow. We focus on teledermatology, where photo quality is particularly important; the framework proposed here can be generalized to other health domains. For telemedicine, dermatologists request that patients submit images of their lesions for assessment. However, these images are often of insufficient quality to make a clinical diagnosis since patients do not have experience taking clinical photos. A clinician has to manually triage poor quality images and request new images to be submitted, leading to wasted time for both the clinician and the patient. We propose an automated image assessment machine learning pipeline, TrueImage, to detect poor quality dermatology photos and to guide patients in taking better photos. Our experiments indicate that TrueImage can reject ~50% of the sub-par quality images, while retaining ~80% of good quality images patients send in, despite heterogeneity and limitations in the training data. These promising results suggest that our solution is feasible and can improve the quality of teledermatology care.