[Artificial intelligence in ophthalmology : Guidelines for physicians for the critical evaluation of studies]. / Künstliche Intelligenz in der Augenheilkunde : Leitfaden für Ärzte zur kritischen Bewertung von Studien.

BACKGROUND: Empirical models have been an integral part of everyday clinical practice in ophthalmology since the introduction of the Sanders-Retzlaff-Kraff (SRK) formula. Recent developments in the field of statistical learning (artificial intelligence, AI) now enable an empirical approach to a wide range of ophthalmological questions with an unprecedented precision. OBJECTIVE: Which criteria must be considered for the evaluation of AI-related studies in ophthalmology? MATERIAL AND METHODS: Exemplary prediction of visual acuity (continuous outcome) and classification of healthy and diseased eyes (discrete outcome) using retrospectively compiled optical coherence tomography data (50 eyes of 50 patients, 50 healthy eyes of 50 subjects). The data were analyzed with nested cross-validation (for learning algorithm selection and hyperparameter optimization). RESULTS: Based on nested cross-validation for training, visual acuity could be predicted in the separate test data-set with a mean absolute error (MAE, 95% confidence interval, CI of 0.142 LogMAR [0.077; 0.207]). Healthy versus diseased eyes could be classified in the test data-set with an agreement of 0.92 (Cohen's kappa). The exemplary incorrect learning algorithm and variable selection resulted in an MAE for visual acuity prediction of 0.229 LogMAR [0.150; 0.309] for the test data-set. The drastic overfitting became obvious on comparison of the MAE with the null model MAE (0.235 LogMAR [0.148; 0.322]). CONCLUSION: Selection of an unsuitable measure of the goodness-of-fit, inadequate validation, or withholding of a null or reference model can obscure the actual goodness-of-fit of AI models. The illustrated pitfalls can help clinicians to identify such shortcomings.

Automated subset identification and characterization pipeline for multidimensional flow and mass cytometry data clustering and visualization.

When examining datasets of any dimensionality, researchers frequently aim to identify individual subsets (clusters) of objects within the dataset. The ubiquity of multidimensional data has motivated the replacement of user-guided clustering with fully automated clustering. The fully automated methods are designed to make clustering more accurate, standardized and faster. However, the adoption of these methods is still limited by the lack of intuitive visualization and cluster matching methods that would allow users to readily interpret fully automatically generated clusters. To address these issues, we developed a fully automated subset identification and characterization (SIC) pipeline providing robust cluster matching and data visualization tools for high-dimensional flow/mass cytometry (and other) data. This pipeline automatically (and intuitively) generates two-dimensional representations of high-dimensional datasets that are safe from the curse of dimensionality. This new approach allows more robust and reproducible data analysis,+ facilitating the development of new gold standard practices across laboratories and institutions.

QFMatch: multidimensional flow and mass cytometry samples alignment.

Part of the flow/mass cytometry data analysis process is aligning (matching) cell subsets between relevant samples. Current methods address this cluster-matching problem in ways that are either computationally expensive, affected by the curse of dimensionality, or fail when population patterns significantly vary between samples. Here, we introduce a quadratic form (QF)-based cluster matching algorithm (QFMatch) that is computationally efficient and accommodates cases where population locations differ significantly (or even disappear or appear) from sample to sample. We demonstrate the effectiveness of QFMatch by evaluating sample datasets from immunology studies. The algorithm is based on a novel multivariate extension of the quadratic form distance for the comparison of flow cytometry data sets. We show that this QF distance has attractive computational and statistical properties that make it well suited for analysis tasks that involve the comparison of flow/mass cytometry samples.

Earth Mover's Distance (EMD): A True Metric for Comparing Biomarker Expression Levels in Cell Populations.

Changes in the frequencies of cell subsets that (co)express characteristic biomarkers, or levels of the biomarkers on the subsets, are widely used as indices of drug response, disease prognosis, stem cell reconstitution, etc. However, although the currently available computational "gating" tools accurately reveal subset frequencies and marker expression levels, they fail to enable statistically reliable judgements as to whether these frequencies and expression levels differ significantly between/among subject groups. Here we introduce flow cytometry data analysis pipeline which includes the Earth Mover's Distance (EMD) metric as solution to this problem. Well known as an informative quantitative measure of differences between distributions, we present three exemplary studies showing that EMD 1) reveals clinically-relevant shifts in two markers on blood basophils responding to an offending allergen; 2) shows that ablative tumor radiation induces significant changes in the murine colon cancer tumor microenvironment; and, 3) ranks immunological differences in mouse peritoneal cavity cells harvested from three genetically distinct mouse strains.

Peer Assessment Enhances Student Learning: The Results of a Matched Randomized Crossover Experiment in a College Statistics Class.

Feedback has a powerful influence on learning, but it is also expensive to provide. In large classes it may even be impossible for instructors to provide individualized feedback. Peer assessment is one way to provide personalized feedback that scales to large classes. Besides these obvious logistical benefits, it has been conjectured that students also learn from the practice of peer assessment. However, this has never been conclusively demonstrated. Using an online educational platform that we developed, we conducted an in-class matched-set, randomized crossover experiment with high power to detect small effects. We establish that peer assessment causes a small but significant gain in student achievement. Our study also demonstrates the potential of web-based platforms to facilitate the design of high-quality experiments to identify small effects that were previously not detectable.

AutoGate: automating analysis of flow cytometry data.

Nowadays, one can hardly imagine biology and medicine without flow cytometry to measure CD4 T cell counts in HIV, follow bone marrow transplant patients, characterize leukemias, etc. Similarly, without flow cytometry, there would be a bleak future for stem cell deployment, HIV drug development and full characterization of the cells and cell interactions in the immune system. But while flow instruments have improved markedly, the development of automated tools for processing and analyzing flow data has lagged sorely behind. To address this deficit, we have developed automated flow analysis software technology, provisionally named AutoComp and AutoGate. AutoComp acquires sample and reagent labels from users or flow data files, and uses this information to complete the flow data compensation task. AutoGate replaces the manual subsetting capabilities provided by current analysis packages with newly defined statistical algorithms that automatically and accurately detect, display and delineate subsets in well-labeled and well-recognized formats (histograms, contour and dot plots). Users guide analyses by successively specifying axes (flow parameters) for data subset displays and selecting statistically defined subsets to be used for the next analysis round. Ultimately, this process generates analysis "trees" that can be applied to automatically guide analyses for similar samples. The first AutoComp/AutoGate version is currently in the hands of a small group of users at Stanford, Emory and NIH. When this "early adopter" phase is complete, the authors expect to distribute the software free of charge to .edu, .org and .gov users.

Automatic clustering of flow cytometry data with density-based merging.

The ability of flow cytometry to allow fast single cell interrogation of a large number of cells has made this technology ubiquitous and indispensable in the clinical and laboratory setting. A current limit to the potential of this technology is the lack of automated tools for analyzing the resulting data. We describe methodology and software to automatically identify cell populations in flow cytometry data. Our approach advances the paradigm of manually gating sequential two-dimensional projections of the data to a procedure that automatically produces gates based on statistical theory. Our approach is nonparametric and can reproduce nonconvex subpopulations that are known to occur in flow cytometry samples, but which cannot be produced with current parametric model-based approaches. We illustrate the methodology with a sample of mouse spleen and peritoneal cavity cells.