Embryo ranking agreement between embryologists and artificial intelligence algorithms.

OBJECTIVE: To evaluate the degree of agreement of embryo ranking between embryologists and eight artificial intelligence (AI) algorithms. DESIGN: Retrospective study. PATIENT(S): A total of 100 cycles with at least eight embryos were selected from the Weill Cornell Medicine database. For each embryo, the full-length time-lapse (TL) videos, as well as a single embryo image at 120 hours, were given to five embryologists and eight AI algorithms for ranking. INTERVENTION(S): None. MAIN OUTCOME MEASURE(S): Kendall rank correlation coefficient (Kendall's τ). RESULT(S): Embryologists had a high degree of agreement in the overall ranking of 100 cycles with an average Kendall's tau (K-τ) of 0.70, slightly lower than the interembryologist agreement when using a single image or video (average K-τ = 0.78). Overall agreement between embryologists and the AI algorithms was significantly lower (average K-τ = 0.53) and similar to the observed low inter-AI algorithm agreement (average K-τ = 0.47). Notably, two of the eight algorithms had a very low agreement with other ranking methodologies (average K-τ = 0.05) and between each other (K-τ = 0.01). The average agreement in selecting the best-quality embryo (1/8 in 100 cycles with an expected agreement by random chance of 12.5%; confidence interval [CI]95: 6%-19%) was 59.5% among embryologists and 40.3% for six AI algorithms. The incidence of the agreement for the two algorithms with the low overall agreement was 11.7%. Agreement on selecting the same top two embryos/cycle (expected agreement by random chance corresponds to 25.0%; CI95: 17%-32%) was 73.5% among embryologists and 56.0% among AI methods excluding two discordant algorithms, which had an average agreement of 24.4%, the expected range of agreement by random chance. Intraembryologist ranking agreement (single image vs. video) was 71.7% and 77.8% for single and top two embryos, respectively. Analysis of average raw scores indicated that cycles with low diversity of embryo quality generally resulted in a lower overall agreement between the methods (embryologists and AI models). CONCLUSION(S): To our knowledge, this is the first study that evaluates the level of agreement in ranking embryo quality between different AI algorithms and embryologists. The different concordance methods were consistent and indicated that the highest agreement was intraembryologist agreement, followed by interembryologist agreement. In contrast, the agreement between some of the AI algorithms and embryologists was similar to the inter-AI algorithm agreement, which also showed a wide range of pairwise concordance. Specifically, two AI models showed intra- and interagreement at the level expected from random selection.

A non-invasive artificial intelligence approach for the prediction of human blastocyst ploidy: a retrospective model development and validation study.

BACKGROUND: One challenge in the field of in-vitro fertilisation is the selection of the most viable embryos for transfer. Morphological quality assessment and morphokinetic analysis both have the disadvantage of intra-observer and inter-observer variability. A third method, preimplantation genetic testing for aneuploidy (PGT-A), has limitations too, including its invasiveness and cost. We hypothesised that differences in aneuploid and euploid embryos that allow for model-based classification are reflected in morphology, morphokinetics, and associated clinical information. METHODS: In this retrospective study, we used machine-learning and deep-learning approaches to develop STORK-A, a non-invasive and automated method of embryo evaluation that uses artificial intelligence to predict embryo ploidy status. Our method used a dataset of 10 378 embryos that consisted of static images captured at 110 h after intracytoplasmic sperm injection, morphokinetic parameters, blastocyst morphological assessments, maternal age, and ploidy status. Independent and external datasets, Weill Cornell Medicine EmbryoScope+ (WCM-ES+; Weill Cornell Medicine Center of Reproductive Medicine, NY, USA) and IVI Valencia (IVI Valencia, Health Research Institute la Fe, Valencia, Spain) were used to test the generalisability of STORK-A and were compared measuring accuracy and area under the receiver operating characteristic curve (AUC). FINDINGS: Analysis and model development included the use of 10 378 embryos, all with PGT-A results, from 1385 patients (maternal age range 21-48 years; mean age 36·98 years [SD 4·62]). STORK-A predicted aneuploid versus euploid embryos with an accuracy of 69·3% (95% CI 66·9-71·5; AUC 0·761; positive predictive value [PPV] 76·1%; negative predictive value [NPV] 62·1%) when using images, maternal age, morphokinetics, and blastocyst score. A second classification task trained to predict complex aneuploidy versus euploidy and single aneuploidy produced an accuracy of 74·0% (95% CI 71·7-76·1; AUC 0·760; PPV 54·9%; NPV 87·6%) using an image, maternal age, morphokinetic parameters, and blastocyst grade. A third classification task trained to predict complex aneuploidy versus euploidy had an accuracy of 77·6% (95% CI 75·0-80·0; AUC 0·847; PPV 76·7%; NPV 78·0%). STORK-A reported accuracies of 63·4% (AUC 0·702) on the WCM-ES+ dataset and 65·7% (AUC 0·715) on the IVI Valencia dataset, when using an image, maternal age, and morphokinetic parameters, similar to the STORK-A test dataset accuracy of 67·8% (AUC 0·737), showing generalisability. INTERPRETATION: As a proof of concept, STORK-A shows an ability to predict embryo ploidy in a non-invasive manner and shows future potential as a standardised supplementation to traditional methods of embryo selection and prioritisation for implantation or recommendation for PGT-A. FUNDING: US National Institutes of Health.