Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine.
J Am Med Inform Assoc
; 31(1): 35-44, 2023 12 22.
Article
em En
| MEDLINE
| ID: mdl-37604111
ABSTRACT
OBJECTIVE:
Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate methods that predict healthcare outcomes. We focused on the prediction of all-cause mortality as the community challenge question. MATERIALS ANDMETHODS:
Using a Model-to-Data framework, 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries, generated 25 accurate models all trained on a dataset of over 1.1 million patients and evaluated on patients prospectively collected over a 1-year observation of a large health system.RESULTS:
The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI, 0.942-0.951) and an area under the precision-recall curve of 0.487 (95% CI, 0.458-0.499) on a prospectively collected patient cohort.DISCUSSION:
Post hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data.CONCLUSION:
This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.Palavras-chave
Texto completo:
1
Base de dados:
MEDLINE
Assunto principal:
Crowdsourcing
/
Medicina
Tipo de estudo:
Prognostic_studies
/
Risk_factors_studies
Limite:
Humans
Idioma:
En
Ano de publicação:
2023
Tipo de documento:
Article