{"doi":"10.1101/2022.09.23.22280295","title":"“E Pluribus Unum”: Prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology (C3RO) Crowdsourced Initiative for Multi-Observer Segmentation","abstract":"Abstract OBJECTIVE Contouring Collaborative for Consensus in Radiation Oncology (C3RO) is a crowdsourced challenge engaging radiation oncologists across various expertise levels in segmentation. A challenge in artificial intelligence (AI) development is the paucity of multi-expert datasets; consequently, we sought to characterize whether aggregate segmentations generated from multiple non-experts could meet or exceed recognized expert agreement. MATERIALS AND METHODS Participants who contoured ≥1 region of interest (ROI) for the breast, sarcoma, head and neck (H&N), gynecologic (GYN), or gastrointestinal (GI) challenge were identified as a non-expert or recognized expert. Cohort-specific ROIs were combined into single simultaneous truth and performance level estimation (STAPLE) consensus segmentations. STAPLE non-expert ROIs were evaluated against STAPLE expert contours using Dice Similarity Coefficient (DSC). The expert interobserver DSC (IODSC expert ) was calculated as an acceptability threshold between STAPLE non-expert and STAPLE expert . To determine the number of non-experts required to match the IODSC expert for each ROI, a single consensus contour was generated using variable numbers of non-experts and then compared to the IODSC expert . RESULTS For all cases, the DSC for STAPLE non-expert versus STAPLE expert were higher than comparator expert IODSC expert for most ROIs. The minimum number of non-expert segmentations needed for a consensus ROI to achieve IODSC expert acceptability criteria ranged between 2-4 for breast, 3-5 for sarcoma, 3-5 for H&N, 3-5 for GYN ROIs, and 3 for GI ROIs. DISCUSSION AND CONCLUSION Multiple non-expert-generated consensus ROIs met or exceeded expert-derived acceptability thresholds. 5 non-experts could potentially generate consensus segmentations for most ROIs with performance approximating experts, suggesting non-expert segmentations as feasible cost-effective AI inputs.","journal":"medRxiv","year":2022,"id":300464,"datarank":0.0,"base_score":0.0,"endowment":0.0,"self_citation_contribution":0.0,"citation_network_contribution":0.0,"self_endowment_contribution":0.0,"citer_contribution":0.0,"corpus_percentile":null,"corpus_rank":null,"citation_count":3,"citer_count":0,"citers_with_citation_signal":0,"citers_with_endowment":0,"datacite_reuse_total":0,"is_dataset":false,"is_dataset_confidence":0.6503,"is_data_producer":false,"deposit_databanks":null,"is_oa":true,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":"2022-01-01","fair_score":null,"fair_percentile":null,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":519105,"name":"Kareem A. Wahid","orcid":"0000-0002-0503-0175","position":1,"is_corresponding":false},{"id":991464,"name":"Benjamin E. Nelms","orcid":null,"position":2,"is_corresponding":false},{"id":426705,"name":"Renjie He","orcid":"0000-0001-9166-6286","position":3,"is_corresponding":false},{"id":653657,"name":"Mohamed A. Naser","orcid":"0000-0003-1020-4966","position":4,"is_corresponding":false},{"id":991465,"name":"Simon Duke","orcid":null,"position":5,"is_corresponding":false},{"id":990954,"name":"Michael V. Sherer","orcid":"0000-0002-4439-7906","position":6,"is_corresponding":false},{"id":654779,"name":"John P. Christodouleas","orcid":"0000-0001-5061-2038","position":7,"is_corresponding":false},{"id":351182,"name":"Abdallah Mohamed","orcid":"0000-0003-2064-7613","position":8,"is_corresponding":false},{"id":990955,"name":"Michael Cislo","orcid":"0000-0002-5880-2802","position":9,"is_corresponding":false},{"id":990956,"name":"James D. Murphy","orcid":"0009-0008-1719-5111","position":10,"is_corresponding":false},{"id":295555,"name":"Clifton D. Fuller","orcid":"0000-0002-5264-3994","position":11,"is_corresponding":false},{"id":278264,"name":"Erin F. Gillespie","orcid":"0000-0002-1386-1542","position":12,"is_corresponding":false},{"id":890779,"name":"Diana Lin","orcid":"0000-0002-1173-0725","position":0,"is_corresponding":true}],"reference_count":35,"raw_metadata":null,"created_at":"2026-07-19T00:31:53.559757Z","pmid":null,"pmcid":null,"fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":null,"license":null,"views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":null,"fair_a":null,"fair_i":null,"fair_r":null,"fair_zscore":null,"fair_rationale":null,"fair_model":null,"fair_agent_version":null,"fair_fulltext_source":null,"fair_has_llm":null,"fair_computed_at":null,"clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}