{"doi":"10.1093/jamia/ocad159","title":"Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine","abstract":"OBJECTIVE: Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate methods that predict healthcare outcomes. We focused on the prediction of all-cause mortality as the community challenge question. MATERIALS AND METHODS: Using a Model-to-Data framework, 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries, generated 25 accurate models all trained on a dataset of over 1.1 million patients and evaluated on patients prospectively collected over a 1-year observation of a large health system. RESULTS: The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI, 0.942-0.951) and an area under the precision-recall curve of 0.487 (95% CI, 0.458-0.499) on a prospectively collected patient cohort. DISCUSSION: Post hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data. CONCLUSION: This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.","journal":"Journal of the American Medical Informatics Association","year":2023,"id":380140,"datarank":0.0,"base_score":0.0,"endowment":0.0,"self_citation_contribution":0.0,"citation_network_contribution":0.0,"self_endowment_contribution":0.0,"citer_contribution":0.0,"corpus_percentile":null,"corpus_rank":null,"citation_count":8,"citer_count":0,"citers_with_citation_signal":0,"citers_with_endowment":0,"datacite_reuse_total":0,"is_dataset":false,"is_dataset_confidence":0.9468,"is_data_producer":false,"deposit_databanks":null,"is_oa":true,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":"2023-01-01","fair_score":null,"fair_percentile":null,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":382617,"name":"Thomas Schaffter","orcid":"0000-0002-8242-9462","position":1,"is_corresponding":false},{"id":382616,"name":"Yao Yan","orcid":"0000-0002-6544-1478","position":2,"is_corresponding":false},{"id":12233,"name":"Thomas Yu","orcid":"0000-0002-5841-0198","position":3,"is_corresponding":false},{"id":384057,"name":"Justin Prosser","orcid":null,"position":4,"is_corresponding":false},{"id":811053,"name":"Jifan Gao","orcid":"0000-0002-6414-130X","position":5,"is_corresponding":false},{"id":340544,"name":"Guanhua Chen","orcid":"0000-0002-9314-2037","position":6,"is_corresponding":false},{"id":811054,"name":"Łukasz Charzewski","orcid":"0000-0002-4667-1390","position":7,"is_corresponding":false},{"id":811728,"name":"Zofia Nawalany","orcid":null,"position":8,"is_corresponding":false},{"id":811055,"name":"Ivan Brugere","orcid":"0000-0002-2953-3746","position":9,"is_corresponding":false},{"id":770658,"name":"Renata Retkutė","orcid":"0000-0002-3877-6440","position":10,"is_corresponding":false},{"id":1145658,"name":"Alisa Prusokiene","orcid":null,"position":11,"is_corresponding":false},{"id":771206,"name":"Augustinas Prusokas","orcid":null,"position":12,"is_corresponding":false},{"id":811056,"name":"Yonghwa Choi","orcid":"0000-0001-5709-9898","position":13,"is_corresponding":false},{"id":811057,"name":"Sang-Hoon Lee","orcid":"0009-0000-9828-1063","position":14,"is_corresponding":false},{"id":811058,"name":"Junseok Choe","orcid":"0000-0001-9548-7146","position":15,"is_corresponding":false},{"id":811729,"name":"Inggeol Lee","orcid":null,"position":16,"is_corresponding":false},{"id":12384,"name":"Sunkyu Kim","orcid":"0000-0002-0240-6210","position":17,"is_corresponding":false},{"id":12234,"name":"Jaewoo Kang","orcid":"0000-0001-6798-9106","position":18,"is_corresponding":false},{"id":40469,"name":"Sean D. Mooney","orcid":"0000-0003-2654-0833","position":19,"is_corresponding":false},{"id":225104,"name":"Justin Guinney","orcid":"0000-0003-1477-1888","position":20,"is_corresponding":false},{"id":86323,"name":"Aaron Lee","orcid":"0000-0002-7452-1648","position":21,"is_corresponding":false},{"id":1145244,"name":"Ali Salehzadeh‐Yazdi","orcid":"0000-0002-1678-0051","position":22,"is_corresponding":false},{"id":771207,"name":"Alidivinas Prusokas","orcid":null,"position":23,"is_corresponding":false},{"id":4252,"name":"Anand Basu","orcid":"0000-0002-9908-9097","position":24,"is_corresponding":false},{"id":951238,"name":"Anas Belouali","orcid":"0000-0002-2780-2500","position":25,"is_corresponding":false},{"id":1145245,"name":"Ann‐Kristin Becker","orcid":"0000-0003-1906-0583","position":26,"is_corresponding":false},{"id":554265,"name":"Ariel Israel","orcid":"0000-0003-4389-8896","position":27,"is_corresponding":false},{"id":1145246,"name":"Benjamin Winter","orcid":"0000-0003-4644-4099","position":29,"is_corresponding":false},{"id":1145659,"name":"Carlos Vega Moreno","orcid":null,"position":30,"is_corresponding":false},{"id":12395,"name":"Christoph Kurz","orcid":"0000-0001-9498-8002","position":31,"is_corresponding":false},{"id":234780,"name":"Dagmar Waltemath","orcid":"0000-0002-5886-5563","position":32,"is_corresponding":false},{"id":1145660,"name":"Darius Schweinoch","orcid":null,"position":33,"is_corresponding":false},{"id":504218,"name":"Enrico Glaab","orcid":"0000-0003-3977-7469","position":34,"is_corresponding":false},{"id":302601,"name":"Gang Luo","orcid":"0000-0001-7217-4008","position":35,"is_corresponding":false},{"id":49735,"name":"Helena U. Zacharias","orcid":"0000-0003-3633-1330","position":37,"is_corresponding":false},{"id":1145247,"name":"Hezhe Qiao","orcid":"0000-0003-3511-0528","position":38,"is_corresponding":false},{"id":1145248,"name":"Julia Truthmann","orcid":"0000-0002-2849-7608","position":43,"is_corresponding":false},{"id":424718,"name":"Kari A. Stephens","orcid":"0000-0002-2520-5526","position":45,"is_corresponding":false},{"id":56054,"name":"Lars Kaderali","orcid":"0000-0002-2359-2294","position":46,"is_corresponding":false},{"id":1145249,"name":"Lav R. Varshney","orcid":"0000-0003-2798-5308","position":47,"is_corresponding":false},{"id":1079642,"name":"Marcus Vollmer","orcid":"0000-0002-4593-2026","position":48,"is_corresponding":false},{"id":1145661,"name":"Maria‐Theodora Pandi","orcid":null,"position":49,"is_corresponding":false},{"id":925769,"name":"Martin L. Gunn","orcid":"0000-0001-9879-8660","position":50,"is_corresponding":false},{"id":296384,"name":"Meliha Yetişgen","orcid":"0000-0001-9919-9811","position":51,"is_corresponding":false},{"id":1145250,"name":"Neetika Nath","orcid":"0000-0002-2156-9576","position":52,"is_corresponding":false},{"id":382618,"name":"Noah Hammarlund","orcid":"0000-0002-3446-2700","position":53,"is_corresponding":false},{"id":1145662,"name":"Oliver Müller-Stricker","orcid":null,"position":54,"is_corresponding":false},{"id":1145663,"name":"Panagiotis Togias","orcid":null,"position":55,"is_corresponding":false},{"id":373971,"name":"Patrick J. Heagerty","orcid":"0000-0002-3403-6007","position":56,"is_corresponding":false},{"id":805330,"name":"Peter Muir","orcid":"0000-0003-2697-0892","position":57,"is_corresponding":false},{"id":504219,"name":"Peter Banda","orcid":"0000-0001-6678-490X","position":58,"is_corresponding":false},{"id":234820,"name":"Ron Henkel","orcid":"0000-0001-6211-2719","position":60,"is_corresponding":false},{"id":1145664,"name":"Sagar Madgi","orcid":null,"position":61,"is_corresponding":false},{"id":951239,"name":"Samir Gupta","orcid":"0000-0003-2656-3912","position":62,"is_corresponding":false},{"id":261660,"name":"Sanghoon Lee","orcid":"0000-0003-2173-6490","position":63,"is_corresponding":false},{"id":1145665,"name":"Shabeeb Kannattikuni","orcid":null,"position":65,"is_corresponding":false},{"id":1145251,"name":"Shamim Sarhadi","orcid":"0000-0002-8379-9916","position":66,"is_corresponding":false},{"id":1145666,"name":"Shikhar Omar","orcid":null,"position":67,"is_corresponding":false},{"id":534777,"name":"Shuo Wang","orcid":"0000-0001-8938-2364","position":68,"is_corresponding":false},{"id":56844,"name":"Soumyabrata Ghosh","orcid":"0000-0003-0659-6733","position":69,"is_corresponding":false},{"id":1145252,"name":"Stefan Neumann","orcid":"0000-0002-3981-1500","position":70,"is_corresponding":false},{"id":1145253,"name":"Stefan Simm","orcid":"0000-0001-9371-2709","position":71,"is_corresponding":false},{"id":225702,"name":"Subha Madhavan","orcid":"0000-0001-7617-3547","position":72,"is_corresponding":false},{"id":35903,"name":"Venkata Satagopam","orcid":"0000-0002-6532-5880","position":75,"is_corresponding":false},{"id":74962,"name":"Vikas Pejaver","orcid":"0000-0002-1943-0284","position":76,"is_corresponding":false},{"id":1145254,"name":"Yachee Gupta","orcid":"0000-0002-2161-2677","position":77,"is_corresponding":false},{"id":49887,"name":"Timothy Bergquist","orcid":"0000-0001-5614-8977","position":0,"is_corresponding":true}],"reference_count":34,"raw_metadata":null,"created_at":"2026-07-19T01:17:00.789848Z","pmid":"37604111","pmcid":null,"fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":null,"license":null,"views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":null,"fair_a":null,"fair_i":null,"fair_r":null,"fair_zscore":null,"fair_rationale":null,"fair_model":null,"fair_agent_version":null,"fair_fulltext_source":null,"fair_has_llm":null,"fair_computed_at":null,"clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}