{"doi":"10.1609/aaai.v33i01.3301590","title":"CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison","abstract":"<jats:p>Large, labeled datasets have driven deep learning methods to achieve expert-level performance on a variety of medical imaging tasks. We present CheXpert, a large dataset that contains 224,316 chest radiographs of 65,240 patients. We design a labeler to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation. We investigate different approaches to using the uncertainty labels for training convolutional neural networks that output the probability of these observations given the available frontal and lateral radiographs. On a validation set of 200 chest radiographic studies which were manually annotated by 3 board-certified radiologists, we find that different uncertainty approaches are useful for different pathologies. We then evaluate our best model on a test set composed of 500 chest radiographic studies annotated by a consensus of 5 board-certified radiologists, and compare the performance of our model to that of 3 additional radiologists in the detection of 5 selected pathologies. On Cardiomegaly, Edema, and Pleural Effusion, the model ROC and PR curves lie above all 3 radiologist operating points. We release the dataset to the public as a standard benchmark to evaluate performance of chest radiograph interpretation models.</jats:p>","journal":"Proceedings of the AAAI Conference on Artificial Intelligence","year":2019,"id":3396,"datarank":13.054715142295626,"base_score":5.971261839790462,"endowment":5.971261839790462,"self_citation_contribution":0.8956892759685695,"citation_network_contribution":12.159025866327056,"self_endowment_contribution":0.8956892759685695,"citer_contribution":12.159025866327056,"corpus_percentile":84.70301057770546,"corpus_rank":189,"citation_count":1710,"citer_count":189,"citers_with_citation_signal":189,"citers_with_endowment":189,"datacite_reuse_total":0,"is_dataset":true,"is_dataset_confidence":0.8689,"is_oa":true,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":"2019-07-17","fair_score":39.5833,"fair_percentile":20.294635004397538,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":35809,"name":"Pranav Rajpurkar","orcid":"0000-0002-8030-3727","position":1,"is_corresponding":false},{"id":35810,"name":"Michael Ko","orcid":"0000-0002-2273-7359","position":2,"is_corresponding":false},{"id":35811,"name":"Yifan Yu","orcid":"0000-0002-3317-1960","position":3,"is_corresponding":false},{"id":35812,"name":"Silviana Ciurea-Ilcus","orcid":null,"position":4,"is_corresponding":false},{"id":35813,"name":"Chris Chute","orcid":null,"position":5,"is_corresponding":false},{"id":35814,"name":"Henrik Marklund","orcid":"0009-0004-5033-6175","position":6,"is_corresponding":false},{"id":35815,"name":"Behzad Haghgoo","orcid":null,"position":7,"is_corresponding":false},{"id":35816,"name":"Robyn Ball","orcid":null,"position":8,"is_corresponding":false},{"id":35817,"name":"Katie Shpanskaya","orcid":"0000-0003-2741-4046","position":9,"is_corresponding":false},{"id":35818,"name":"Jayne Seekins","orcid":"0000-0002-5698-0840","position":10,"is_corresponding":false},{"id":35819,"name":"David A. Mong","orcid":"0000-0003-0279-0301","position":11,"is_corresponding":false},{"id":35820,"name":"Safwan S. Halabi","orcid":"0000-0003-1317-984X","position":12,"is_corresponding":false},{"id":35821,"name":"Jesse K. Sandberg","orcid":"0000-0001-9980-8859","position":13,"is_corresponding":false},{"id":96708,"name":"Richard Hayden Jones","orcid":null,"position":14,"is_corresponding":false},{"id":35823,"name":"David B. Larson","orcid":"0000-0002-1157-5905","position":15,"is_corresponding":false},{"id":11422,"name":"Curtis P. Langlotz","orcid":"0000-0002-8972-8051","position":16,"is_corresponding":false},{"id":35824,"name":"Bhavik N. Patel","orcid":"0000-0001-5157-9903","position":17,"is_corresponding":false},{"id":35825,"name":"Matthew P. Lungren","orcid":"0000-0002-8591-5861","position":18,"is_corresponding":false},{"id":11423,"name":"Andrew Y. Ng","orcid":"0000-0001-5547-3196","position":19,"is_corresponding":false},{"id":35826,"name":"Robyn L. Ball","orcid":"0000-0002-7335-3339","position":20,"is_corresponding":false},{"id":35827,"name":"Richard H. Jones","orcid":"0000-0002-7570-2141","position":21,"is_corresponding":false},{"id":35808,"name":"Jeremy Irvin","orcid":"0000-0002-0395-4403","position":0,"is_corresponding":true}],"reference_count":39,"raw_metadata":{"citation_network_status":"fetched"},"created_at":"2026-03-01T18:20:47.508186Z","pmid":null,"pmcid":null,"fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":null,"license":null,"views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":32.5,"fair_a":67.5,"fair_i":25.0,"fair_r":33.3333,"fair_zscore":-0.5091,"fair_rationale":{"fair_score":39.58,"has_llm":true,"dimensions":{"F":{"name":"Findable","score":32.5,"criteria":[{"key":"f_has_doi","label":"Has a persistent DOI","kind":"deterministic","weight":1.0,"fraction":1.0,"signal":"DOI present","rationale":null},{"key":"f_repository_presence","label":"Indexed in repositories / literature DBs","kind":"deterministic","weight":1.0,"fraction":0.0,"signal":"datacite=0, pmcid=False, pmid=False","rationale":null},{"key":"f_persistent_ids","label":"Resolvable scholarly identifiers (OpenAlex)","kind":"deterministic","weight":0.5,"fraction":0.0,"signal":"no OpenAlex id","rationale":null},{"key":"f_metadata_richness","label":"Rich, machine-readable metadata","kind":"llm","weight":1.0,"fraction":0.25,"signal":null,"rationale":"The paper provides descriptions of the dataset (e.g., number of radiographs, patient count, label categories) and a link to the dataset, but it does not provide machine-readable metadata (e.g., structured schema, JSON-LD, or RDF) that would enable automated discovery."}]},"A":{"name":"Accessible","score":67.5,"criteria":[{"key":"a_open_access","label":"Open Access / files deposited","kind":"deterministic","weight":1.5,"fraction":1.0,"signal":"Open Access","rationale":null},{"key":"a_retrievable","label":"Free full text retrievable","kind":"deterministic","weight":1.0,"fraction":0.0,"signal":"0 OA location(s)","rationale":null},{"key":"a_access_protocol","label":"Clear data/code access protocol","kind":"llm","weight":1.0,"fraction":0.75,"signal":null,"rationale":"The paper states the dataset is 'publicly available' and provides a URL (https://stanfordmlgroup.github.io/competitions/chexpert), but it does not describe any authentication, licensing terms, or download protocol in detail."}]},"I":{"name":"Interoperable","score":25.0,"criteria":[{"key":"i_linked_data","label":"Linked datasets / DataCite relations","kind":"deterministic","weight":1.0,"fraction":0.0,"signal":"linked_datasets=0, datacite=0","rationale":null},{"key":"i_standard_ids","label":"References data via standard accessions","kind":"deterministic","weight":1.0,"fraction":0.0,"signal":"accessions=0, trials=0","rationale":null},{"key":"i_standards","label":"Standard formats, vocabularies & identifiers","kind":"llm","weight":1.0,"fraction":0.5,"signal":null,"rationale":"The paper uses standard DICOM-like image formats (implied by radiographs) and standard neural network architectures (DenseNet121), but does not specify use of standard ontologies, controlled vocabularies (beyond the Fleischner Society glossary), or persistent identifiers for data elements."}]},"R":{"name":"Reusable","score":33.33,"criteria":[{"key":"r_license","label":"Clear, open reuse license","kind":"deterministic","weight":1.5,"fraction":0.0,"signal":"no license","rationale":null},{"key":"r_downloads","label":"Demonstrated reuse (downloads)","kind":"deterministic","weight":0.5,"fraction":0.0,"signal":"downloads=0","rationale":null},{"key":"r_version","label":"Versioned / maintained","kind":"deterministic","weight":0.5,"fraction":0.0,"signal":"no version chain","rationale":null},{"key":"r_dataset","label":"Classified as a data resource","kind":"deterministic","weight":0.5,"fraction":1.0,"signal":"is_dataset","rationale":null},{"key":"r_reusability","label":"Data-availability statement, license & reproducibility","kind":"llm","weight":2.0,"fraction":0.5,"signal":null,"rationale":"The paper states the dataset is released publicly and provides a benchmark, but it lacks an explicit data-availability statement, a license, and does not detail code or model reproducibility steps for the analysis."}]}},"suggestions":["Include a machine-readable metadata file (e.g., JSON-LD or schema.org) describing the dataset's structure, provenance, and links.","Specify an explicit license (e.g., CC BY 4.0) and mention the terms of use in the paper or dataset landing page.","Use persistent identifiers (e.g., DOIs) for the dataset and provide them in the paper.","Publish the model training code and configuration files (e.g., Docker environment, hyperparameters) to enable full reproducibility.","Provide a data dictionary that maps observation labels to standard terms from an ontology (e.g., RadLex or SNOMED CT)."],"model":"deepseek/deepseek-v4-flash","agent_version":"fair_agent_v2","fulltext_source":"unpaywall_pdf"},"fair_model":"deepseek/deepseek-v4-flash","fair_agent_version":"fair_agent_v2","fair_fulltext_source":"unpaywall_pdf","fair_has_llm":true,"fair_computed_at":"2026-06-18T00:31:08.949153Z","clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}