DNA Methylation Signatures Correlate with Response to Immune Checkpoint Inhibitors in Metastatic Melanoma.

BACKGROUND: DNA methylation profiles have emerged as potential predictors of therapeutic response in various solid tumors. OBJECTIVE: This study aimed to analyze the DNA methylation profiles of patients with stage IV metastatic melanoma undergoing first-line immune checkpoint inhibitor treatment and evaluate their correlation with a radiological response according to immune-related Response Evaluation Criteria in Solid Tumors (iRECIST). METHODS: A total of 81 tissue samples from 71 patients with metastatic melanoma (27 female, 44 male) were included in this study. We utilized Illumina Methylation EPIC Beadchips to retrieve their genome-wide methylation profile by interrogating >850,000 CpG sites. Clustering based on the 500 most differentially methylated genes was conducted to identify distinct methylation patterns associated with immune checkpoint inhibitor response. Results were further aligned with an independent, previously published data set. RESULTS: The median progression-free survival was 8.5 months (range: 0-104.1 months), and the median overall survival was 30.6 months (range: 0-104.1 months). Objective responses were observed in 29 patients (40.8%). DNA methylation profiling revealed specific signatures that correlated with radiological response to immune checkpoint inhibitors. Three distinct clusters were identified based on the methylation patterns of the 500 most differentially methylated genes. Cluster 1 (12/12) and cluster 2 (12/24) exhibited a higher proportion of responders, while cluster 3 (39/45) predominantly consisted of non-responders. In the validation data set, responders also showed more frequent hypomethylation although differences in the data sets limit the interpretation. CONCLUSIONS: These findings suggest that DNA methylation profiling of tumor tissues might serve as a predictive biomarker for immune checkpoint inhibitor response in patients with metastatic melanoma. Further validation studies are warranted to confirm the efficiency of DNA methylation profiling as a predictive tool in the context of immunotherapy for metastatic melanoma.

Dermatologist-like explainable AI enhances trust and confidence in diagnosing melanoma.

Artificial intelligence (AI) systems have been shown to help dermatologists diagnose melanoma more accurately, however they lack transparency, hindering user acceptance. Explainable AI (XAI) methods can help to increase transparency, yet often lack precise, domain-specific explanations. Moreover, the impact of XAI methods on dermatologists' decisions has not yet been evaluated. Building upon previous research, we introduce an XAI system that provides precise and domain-specific explanations alongside its differential diagnoses of melanomas and nevi. Through a three-phase study, we assess its impact on dermatologists' diagnostic accuracy, diagnostic confidence, and trust in the XAI-support. Our results show strong alignment between XAI and dermatologist explanations. We also show that dermatologists' confidence in their diagnoses, and their trust in the support system significantly increase with XAI compared to conventional AI. This study highlights dermatologists' willingness to adopt such XAI systems, promoting future use in the clinic.

Position statement of the EADV Artificial Intelligence (AI) Task Force on AI-assisted smartphone apps and web-based services for skin disease.

BACKGROUND: As the use of smartphones continues to surge globally, mobile applications (apps) have become a powerful tool for healthcare engagement. Prominent among these are dermatology apps powered by Artificial Intelligence (AI), which provide immediate diagnostic guidance and educational resources for skin diseases, including skin cancer. OBJECTIVE: This article, authored by the EADV AI Task Force, seeks to offer insights and recommendations for the present and future deployment of AI-assisted smartphone applications (apps) and web-based services for skin diseases with emphasis on skin cancer detection. METHODS: An initial position statement was drafted on a comprehensive literature review, which was subsequently refined through two rounds of digital discussions and meticulous feedback by the EADV AI Task Force, ensuring its accuracy, clarity and relevance. RESULTS: Eight key considerations were identified, including risks associated with inaccuracy and improper user education, a decline in professional skills, the influence of non-medical commercial interests, data security, direct and indirect costs, regulatory approval and the necessity of multidisciplinary implementation. Following these considerations, three main recommendations were formulated: (1) to ensure user trust, app developers should prioritize transparency in data quality, accuracy, intended use, privacy and costs; (2) Apps and web-based services should ensure a uniform user experience for diverse groups of patients; (3) European authorities should adopt a rigorous and consistent regulatory framework for dermatology apps to ensure their safety and accuracy for users. CONCLUSIONS: The utilisation of AI-assisted smartphone apps and web-based services in diagnosing and treating skin diseases has the potential to greatly benefit patients in their dermatology journeys. By prioritising innovation, fostering collaboration and implementing effective regulations, we can ensure the successful integration of these apps into clinical practice.

Expert Agreement on the Presence and Spatial Localization of Melanocytic Features in Dermoscopy.

Dermoscopy aids in melanoma detection; however, agreement on dermoscopic features, including those of high clinical relevance, remains poor. In this study, we attempted to evaluate agreement among experts on exemplar images not only for the presence of melanocytic-specific features but also for spatial localization. This was a cross-sectional, multicenter, observational study. Dermoscopy images exhibiting at least 1 of 31 melanocytic-specific features were submitted by 25 world experts as exemplars. Using a web-based platform that allows for image markup of specific contrast-defined regions (superpixels), 20 expert readers annotated 248 dermoscopic images in collections of 62 images. Each collection was reviewed by five independent readers. A total of 4,507 feature observations were performed. Good-to-excellent agreement was found for 14 of 31 features (45.2%), with eight achieving excellent agreement (Gwet's AC >0.75) and seven of them being melanoma-specific features. These features were peppering/granularity (0.91), shiny white streaks (0.89), typical pigment network (0.83), blotch irregular (0.82), negative network (0.81), irregular globules (0.78), dotted vessels (0.77), and blue-whitish veil (0.76). By utilizing an exemplar dataset, a good-to-excellent agreement was found for 14 features that have previously been shown useful in discriminating nevi from melanoma. All images are public (www.isic-archive.com) and can be used for education, scientific communication, and machine learning experiments.

Comparison of humans versus mobile phone-powered artificial intelligence for the diagnosis and management of pigmented skin cancer in secondary care: a multicentre, prospective, diagnostic, clinical trial.

BACKGROUND: Diagnosis of skin cancer requires medical expertise, which is scarce. Mobile phone-powered artificial intelligence (AI) could aid diagnosis, but it is unclear how this technology performs in a clinical scenario. Our primary aim was to test in the clinic whether there was equivalence between AI algorithms and clinicians for the diagnosis and management of pigmented skin lesions. METHODS: In this multicentre, prospective, diagnostic, clinical trial, we included specialist and novice clinicians and patients from two tertiary referral centres in Australia and Austria. Specialists had a specialist medical qualification related to diagnosing and managing pigmented skin lesions, whereas novices were dermatology junior doctors or registrars in trainee positions who had experience in examining and managing these lesions. Eligible patients were aged 18-99 years and had a modified Fitzpatrick I-III skin type; those in the diagnostic trial were undergoing routine excision or biopsy of one or more suspicious pigmented skin lesions bigger than 3 mm in the longest diameter, and those in the management trial had baseline total-body photographs taken within 1-4 years. We used two mobile phone-powered AI instruments incorporating a simple optical attachment: a new 7-class AI algorithm and the International Skin Imaging Collaboration (ISIC) AI algorithm, which was previously tested in a large online reader study. The reference standard for excised lesions in the diagnostic trial was histopathological examination; in the management trial, the reference standard was a descending hierarchy based on histopathological examination, comparison of baseline total-body photographs, digital monitoring, and telediagnosis. The main outcome of this study was to compare the accuracy of expert and novice diagnostic and management decisions with the two AI instruments. Possible decisions in the management trial were dismissal, biopsy, or 3-month monitoring. Decisions to monitor were considered equivalent to dismissal (scenario A) or biopsy of malignant lesions (scenario B). The trial was registered at the Australian New Zealand Clinical Trials Registry ACTRN12620000695909 (Universal trial number U1111-1251-8995). FINDINGS: The diagnostic study included 172 suspicious pigmented lesions (84 malignant) from 124 patients and the management study included 5696 pigmented lesions (18 malignant) from the whole body of 66 high-risk patients. The diagnoses of the 7-class AI algorithm were equivalent to the specialists' diagnoses (absolute accuracy difference 1·2% [95% CI -6·9 to 9·2]) and significantly superior to the novices' ones (21·5% [13·1 to 30·0]). The diagnoses of the ISIC AI algorithm were significantly inferior to the specialists' diagnoses (-11·6% [-20·3 to -3·0]) but significantly superior to the novices' ones (8·7% [-0·5 to 18·0]). The best 7-class management AI was significantly inferior to specialists' management (absolute accuracy difference in correct management decision -0·5% [95% CI -0·7 to -0·2] in scenario A and -0·4% [-0·8 to -0·05] in scenario B). Compared with the novices' management, the 7-class management AI was significantly inferior (-0·4% [-0·6 to -0·2]) in scenario A but significantly superior (0·4% [0·0 to 0·9]) in scenario B. INTERPRETATION: The mobile phone-powered AI technology is simple, practical, and accurate for the diagnosis of suspicious pigmented skin cancer in patients presenting to a specialist setting, although its usage for management decisions requires more careful execution. An AI algorithm that was superior in experimental studies was significantly inferior to specialists in a real-world scenario, suggesting that caution is needed when extrapolating results of experimental studies to clinical practice. FUNDING: MetaOptima Technology.

Development of a localization-based algorithm for the prediction of leg ulcer etiology.

BACKGROUND: Diagnostic work-up of leg ulcers is time- and cost-intensive. This study aimed at evaluating ulcer location as a diagnostic criterium and providing a diagnostic algorithm to facilitate differential diagnosis. PATIENTS AND METHODS: The study consisted of 277 patients with lower leg ulcers. The following five groups were defined: Venous leg ulcer, arterial ulcers, mixed ulcer, arteriolosclerosis, and vasculitis. Using computational surface rendering, predilection sites of different ulcer types were evaluated. The results were integrated in a multinomial logistic regression model to calculate the likelihood of a specific diagnosis depending on location, age, bilateral involvement, and ulcer count. Additionally, neural network image analysis was performed. RESULTS: The majority of venous ulcers extended to the medial malleolar region. Arterial ulcers were most frequently located on the dorsal aspect of the forefoot. Arteriolosclerotic ulcers were distinctly localized at the middle third of the lower leg. Vasculitic ulcers appeared to be randomly distributed and were markedly smaller, multilocular and bilateral. The multinomial logistic regression model showed an overall satisfactory performance with an estimated accuracy of 0.68 on unseen data. CONCLUSIONS: The presented algorithm based on ulcer location may serve as a basic tool to narrow down potential diagnoses and guide further diagnostic work-up.

Delphi Consensus Among International Experts on the Diagnosis, Management, and Surveillance for Lentigo Maligna.

INTRODUCTION: Melanoma of the lentigo maligna (LM) type is challenging. There is lack of consensus on the optimal diagnosis, treatment, and follow-up. OBJECTIVES: To obtain general consensus on the diagnosis, treatment, and follow-up for LM. METHODS: A modified Delphi method was used. The invited participants were either members of the International Dermoscopy Society, academic experts, or authors of published articles relating to skin cancer and melanoma. Participants were required to respond across three rounds using a 4-point Likert scale). Consensus was defined as >75% of participants agreeing/strongly agreeing or disagreeing/strongly disagreeing. RESULTS: Of the 31 experts invited to participate in this Delphi study, 29 participants completed Round 1 (89.9% response rate), 25/31 completed Round 2 (77.5% response rate), and 25/31 completed Round 3 (77.5% response rate). Experts agreed that LM diagnosis should be based on a clinical and dermatoscopic approach (92%) followed by a biopsy. The most appropriate primary treatment of LM was deemed to be margin-controlled surgery (83.3%), although non-surgical modalities, especially imiquimod, were commonly used either as alternative off-label primary treatment in selected patients or as adjuvant therapy following surgery; 62% participants responded life-long clinical follow-up was needed for LM. CONCLUSIONS: Clinical and histological diagnosis of LM is challenging and should be based on macroscopic, dermatoscopic, and RCM examination followed by a biopsy. Different treatment modalities and follow-up should be carefully discussed with the patient.

A reinforcement learning model for AI-based decision support in skin cancer.

We investigated whether human preferences hold the potential to improve diagnostic artificial intelligence (AI)-based decision support using skin cancer diagnosis as a use case. We utilized nonuniform rewards and penalties based on expert-generated tables, balancing the benefits and harms of various diagnostic errors, which were applied using reinforcement learning. Compared with supervised learning, the reinforcement learning model improved the sensitivity for melanoma from 61.4% to 79.5% (95% confidence interval (CI): 73.5-85.6%) and for basal cell carcinoma from 79.4% to 87.1% (95% CI: 80.3-93.9%). AI overconfidence was also reduced while simultaneously maintaining accuracy. Reinforcement learning increased the rate of correct diagnoses made by dermatologists by 12.0% (95% CI: 8.8-15.1%) and improved the rate of optimal management decisions from 57.4% to 65.3% (95% CI: 61.7-68.9%). We further demonstrated that the reward-adjusted reinforcement learning model and a threshold-based model outperformed naïve supervised learning in various clinical scenarios. Our findings suggest the potential for incorporating human preferences into image-based diagnostic algorithms.

Evaluating a CDS Hook for FHIR Questionnaires in a SMART on FHIR App and an Existing Dermatological CDS System.

BACKGROUND: To deploy clinical decision support (CDS) systems in routine patient care they have to be certified as a medical device. The European Medical Device Regulation explicitly asks for the use of standards and interoperability in the approval process. OBJECTIVES: We extended an existing dermatological CDS system with emerging standards for CDS interoperability, to facilitate a future integration into existing healthcare infrastructure, and approval as a medical device. METHODS: The data collection part of a CDS system was extended with the endpoints required by the CDS Hooks specification. FHIR QuestionnaireResponse resources trigger a newly defined hook. RESULTS: One hundred and seventeen clinical observations and patient variables needed for the ranking of a disease were mapped to SNOMED CT or LOINC and modeled as FHIR Questionnaire which is rendered using LHC LForms in a SMART on FHIR app with the SMART Dev Sandbox. CONCLUSION: SMART on FHIR in combination with CDS Hooks facilitates the integration of existing CDS systems into EHR systems, potentially improving education and patient care.

Single-cell RNA sequencing analysis of a COVID-19-associated maculopapular rash in a patient with psoriasis treated with ustekinumab.

Coronavirus disease 2019 (COVID-19) primarily affects the respiratory system but extrapulmonary manifestations, including the skin, have been well documented. However, transcriptomic profiles of skin lesions have not been performed thus far. Here, we present a single-cell RNA sequencing analysis in a patient with COVID-19 infection with a maculopapular skin rash while on treatment with the interleukin (IL)-12/IL-23 blocker ustekinumab for his underlying psoriasis. Results were compared with healthy controls and untreated psoriasis lesions. We found the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) viral entry receptors ACE2 and TMPRSS2 in keratinocytes of the patient with COVID-19, while ACE2 expression was low to undetectable in psoriasis lesions and healthy skin. Among all cell types, ACE2+ keratinocyte clusters showed the highest levels of transcriptomic dysregulation in COVID-19, expressing type 1-associated immune markers such as CXCL9 and CXCL10. In line with a generally type 1-skewed immune microenvironment, cytotoxic lymphocytes showed increased expression of the IFNG gene and other T-cell effector genes, while type 2, type 17, or type 22 T-cell activation was largely absent. Conversely, downregulation of several anti-inflammatory mediators was observed. This first transcriptomic description of a COVID-19-associated rash identifies ACE2+ keratinocytes displaying profound transcriptional changes, and inflammatory immune cells that might help to improve the understanding of SARS-CoV-2-associated skin conditions.

Integrated diagnosis of nevi with severely atypical features and impact of second opinions.

BACKGROUND: A subset of melanocytic proliferations is difficult to classify by dermatopathology alone and their management is challenging. OBJECTIVE: To explore the value of correlation with dermatoscopy and to evaluate the utility of second opinions by additional pathologists. METHODS: For this single center retrospective study we collected 122 lesions that were diagnosed as atypical melanocytic proliferations, we reviewed dermatoscopy and asked two experienced pathologists to reassess the slides independently. RESULTS: For the binary decision of nevus versus melanoma the diagnostic consensus among external pathologists was only moderate (kappa 0.43; 95% CI 0.25-0.61). If ground truth were defined such that both pathologists had to agree on the diagnosis of melanoma, 13.1% of cases would have been diagnosed as melanoma. If one pathologist were sufficient to call it melanoma 29.5% of cases would have been diagnosed as melanoma. In either case, the presence of dermatoscopic white lines was associated with the diagnosis of melanoma. In lesions with peripheral dots and clods, melanoma was not jointly diagnosed by the two pathologists if the patient was younger than 45 years. CONCLUSIONS: A considerable number of atypical melanocytic proliferations may be diagnosed as melanoma if revised by other pathologists. The presence of white lines on dermatoscopy increases the likelihood of revision towards melanoma. Peripheral clods indicate growth but are not a melanoma clue if patients are younger than 45 years.

ORF1p Is a Potential Novel Diagnostic Marker for Differentiated Vulvar Intraepithelial Neoplasia.

Accurate diagnosis of differentiated vulvar intraepithelial neoplasia (dVIN) can be challenging as histomorphologic features may be subtle and overlap with nondysplastic lesions. In practice, aberrant p53 expression supports the diagnosis, but a substantial percentage retains wild-type p53. Recently, the retrotransposon long interspersed nuclear element 1 has been detected in distinct cancer types. We have now investigated the expression of the long interspersed nuclear element 1 encoded protein ORF1p in dysplastic and nondysplastic vulvar samples to assess its diagnostic value. Specimens of dVIN (n=29), high-grade squamous intraepithelial lesions (n=26), inflammatory vulvar lesions (n=20), lichen sclerosus (n=22), and normal vulvar epithelia (n=29) were included. ORF1p and p53 expression was determined using immunohistochemistry. The majority of dVIN [27/29 (93%)] and high-grade squamous intraepithelial lesions [20/26 (77%)] showed distinct (i.e. moderate or strong) ORF1p expression in the basal and suprabasal or all epithelial layers, respectively. Of note, ORF1p was present in all 4 cases of dVIN with wild-type p53 staining pattern. In contrast, ORF1p was negative or weakly expressed in most inflammatory lesions [14/20 (70%)] and lichen sclerosus [18/22 (82%), P <0.001]. Normal control epithelium exhibited negative staining in all cases. In conclusion, ORF1p might be a useful diagnostic marker for dVIN, especially in cases with retained wild-type p53.

Indications for Digital Monitoring of Patients With Multiple Nevi: Recommendations from the International Dermoscopy Society.

Introduction: In patients with multiple nevi, sequential imaging using total body skin photography (TBSP) coupled with digital dermoscopy (DD) documentation reduces unnecessary excisions and improves the early detection of melanoma. Correct patient selection is essential for optimizing the efficacy of this diagnostic approach. Objectives: The purpose of the study was to identify, via expert consensus, the best indications for TBSP and DD follow-up. Methods: This study was performed on behalf of the International Dermoscopy Society (IDS). We attained consensus by using an e-Delphi methodology. The panel of participants included international experts in dermoscopy. In each Delphi round, experts were asked to select from a list of indications for TBSP and DD. Results: Expert consensus was attained after 3 rounds of Delphi. Participants considered a total nevus count of 60 or more nevi or the presence of a CDKN2A mutation sufficient to refer the patient for digital monitoring. Patients with more than 40 nevi were only considered an indication in case of personal history of melanoma or red hair and/or a MC1R mutation or history of organ transplantation. Conclusions: Our recommendations support clinicians in choosing appropriate follow-up regimens for patients with multiple nevi and in applying the time-consuming procedure of sequential imaging more efficiently. Further studies and real-life data are needed to confirm the usefulness of this list of indications in clinical practice.

Application of an Interactive Diagnosis Ranking Algorithm in a Simulated Vignette-based Environment for General Dermatology.

Introduction: Diagnostic algorithms may reduce noise and bias and improve interrater agreement of clinical decisions. In a practical sense, algorithms may serve as alternatives to specialist consultations or decision support in store-and-forward tele-dermatology. It is, however, unknown how dermatologists interact with algorithms based on questionnaires. Objectives: To evaluate the performance of a questionnaire-based diagnostic algorithm when applied by users with different expertise. Methods: We created 58 virtual test cases covering common dermatologic diseases and asked five raters with different expertise to complete a predefined clinical questionnaire, which served as input for a disease ranking algorithm. We compared the ranks of the correct diagnosis between users, analyzed the similarity between inputs of different users, and explored the impact of different parts of the questionnaire on the final ranking. Results: When applied by a board-certified dermatologist, the algorithm top-ranked the correct diagnosis in the majority of cases (median rank 1; interquartile range: 1.0; mean reciprocal rank 0.757). The median rank of the correct diagnosis was significantly lower when the algorithm was applied by four dermatology residents (median rank 2-5, P < 0.01). The lowest similarity between inputs of the residents and the board-certified dermatologist was found for questions regarding morphology. Sensitivity analysis showed the highest deterioration in performance after omission of information on morphology and anatomic site. Conclusions: A simple questionnaire-based disease ranking algorithm provides accurate ranking for a wide variety of dermatologic conditions. When applied in clinical practice, additional measures may be needed to ensure robustness of data entry for inexperienced users.

Comparison of Convolutional Neural Network Architectures for Robustness Against Common Artefacts in Dermatoscopic Images.

Introduction: Classification of dermatoscopic images via neural networks shows comparable performance to clinicians in experimental conditions but can be affected by artefacts like skin markings or rulers. It is unknown whether specialized neural networks are more robust to artefacts. Objectives: Analyze robustness of 3 neural network architectures, namely ResNet-34, Faster R-CNN and Mask R-CNN. Methods: We identified common artefacts in the HAM10000, PH2 and the 7-point criteria evaluation datasets, and established a template-based method to superimpose artefacts on dermatoscopic images. The HAM10000-dataset with and without superimposed artefacts was used to train the networks, followed by analyzing their robustness against artefacts in test images. Performance was assessed via area under the precision recall curve and classification results. Results: ResNet-34 and Faster R-CNN models trained on regular images perform worse than Mask R-CNN on images with superimposed artefacts. Artefacts added to all tested images led to a decrease in area under the precision-recall curve values of 0.030 for ResNet-34 and 0.045 for Faster R-CNN in comparison to only 0.011 for Mask R-CNN. However, changes in model performance only became significant with 40% or more of the images having superimposed artefacts. A loss in performance occurred when the training was biased by selectively superimposing artefacts on images belonging to a certain class. Conclusions: As Mask R-CNN showed the least decrease in performance when confronted with artefacts, instance segmentation architectures may be helpful to counter the effects of artefacts, warranting further research on related architectures. Our artefact insertion mechanism could be useful for future research.

Validation of artificial intelligence prediction models for skin cancer diagnosis using dermoscopy images: the 2019 International Skin Imaging Collaboration Grand Challenge.

BACKGROUND: Previous studies of artificial intelligence (AI) applied to dermatology have shown AI to have higher diagnostic classification accuracy than expert dermatologists; however, these studies did not adequately assess clinically realistic scenarios, such as how AI systems behave when presented with images of disease categories that are not included in the training dataset or images drawn from statistical distributions with significant shifts from training distributions. We aimed to simulate these real-world scenarios and evaluate the effects of image source institution, diagnoses outside of the training set, and other image artifacts on classification accuracy, with the goal of informing clinicians and regulatory agencies about safety and real-world accuracy. METHODS: We designed a large dermoscopic image classification challenge to quantify the performance of machine learning algorithms for the task of skin cancer classification from dermoscopic images, and how this performance is affected by shifts in statistical distributions of data, disease categories not represented in training datasets, and imaging or lesion artifacts. Factors that might be beneficial to performance, such as clinical metadata and external training data collected by challenge participants, were also evaluated. 25 331 training images collected from two datasets (in Vienna [HAM10000] and Barcelona [BCN20000]) between Jan 1, 2000, and Dec 31, 2018, across eight skin diseases, were provided to challenge participants to design appropriate algorithms. The trained algorithms were then tested for balanced accuracy against the HAM10000 and BCN20000 test datasets and data from countries not included in the training dataset (Turkey, New Zealand, Sweden, and Argentina). Test datasets contained images of all diagnostic categories available in training plus other diagnoses not included in training data (not trained category). We compared the performance of the algorithms against that of 18 dermatologists in a simulated setting that reflected intended clinical use. FINDINGS: 64 teams submitted 129 state-of-the-art algorithm predictions on a test set of 8238 images. The best performing algorithm achieved 58·8% balanced accuracy on the BCN20000 data, which was designed to better reflect realistic clinical scenarios, compared with 82·0% balanced accuracy on HAM10000, which was used in a previously published benchmark. Shifted statistical distributions and disease categories not included in training data contributed to decreases in accuracy. Image artifacts, including hair, pen markings, ulceration, and imaging source institution, decreased accuracy in a complex manner that varied based on the underlying diagnosis. When comparing algorithms to expert dermatologists (2460 ratings on 1269 images), algorithms performed better than experts in most categories, except for actinic keratoses (similar accuracy on average) and images from categories not included in training data (26% correct for experts vs 6% correct for algorithms, p<0·0001). For the top 25 submitted algorithms, 47·1% of the images from categories not included in training data were misclassified as malignant diagnoses, which would lead to a substantial number of unnecessary biopsies if current state-of-the-art AI technologies were clinically deployed. INTERPRETATION: We have identified specific deficiencies and safety issues in AI diagnostic systems for skin cancer that should be addressed in future diagnostic evaluation protocols to improve safety and reliability in clinical practice. FUNDING: Melanoma Research Alliance and La Marató de TV3.