Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine.

OBJECTIVE: Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate methods that predict healthcare outcomes. We focused on the prediction of all-cause mortality as the community challenge question. MATERIALS AND METHODS: Using a Model-to-Data framework, 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries, generated 25 accurate models all trained on a dataset of over 1.1 million patients and evaluated on patients prospectively collected over a 1-year observation of a large health system. RESULTS: The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI, 0.942-0.951) and an area under the precision-recall curve of 0.487 (95% CI, 0.458-0.499) on a prospectively collected patient cohort. DISCUSSION: Post hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data. CONCLUSION: This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.

Effectiveness of glass beads for plating cell cultures.

Cell plating, the spreading out of a liquid suspension of cells on a surface followed by colony growth, is a common laboratory procedure in microbiology. Despite this, the exact impact of its parameters on colony growth has not been extensively studied. A common protocol involves the shaking of glass beads within a Petri dish containing solid growth media. We investigated the effects of multiple parameters in this protocol: the number of beads, the shape of movement, and the number of movements. Standard suspensions of Escherichia coli were spread while varying these parameters to assess their impact on colony growth. Results were assessed by a variety of metrics: the number of colonies, the mean distance between closest colonies, and the variability and uniformity of their spatial distribution. Finally, we devised a mathematical model of shifting billiard to explain the heterogeneities in the observed spatial patterns. Exploring the parameters that affect the most fundamental techniques in microbiology allows us to better understand their function, giving us the ability to precisely control their outputs for our exact needs.