{"doi":"10.1038/s41592-021-01336-8","title":"Benchmarking atlas-level data integration in single-cell genomics","abstract":"Single-cell atlases often include samples that span locations, laboratories and conditions, leading to complex, nested batch effects in data. Thus, joint analysis of atlas datasets requires reliable data integration. To guide integration method choice, we benchmarked 68 method and preprocessing combinations on 85 batches of gene expression, chromatin accessibility and simulation data from 23 publications, altogether representing >1.2 million cells distributed in 13 atlas-level integration tasks. We evaluated methods according to scalability, usability and their ability to remove batch effects while retaining biological variation using 14 evaluation metrics. We show that highly variable gene selection improves the performance of data integration methods, whereas scaling pushes methods to prioritize batch removal over conservation of biological variation. Overall, scANVI, Scanorama, scVI and scGen perform well, particularly on complex integration tasks, while single-cell ATAC-sequencing integration performance is strongly affected by choice of feature space. Our freely available Python module and benchmarking pipeline can identify optimal data integration methods for new data, benchmark new methods and improve method development.","journal":"Nature Methods","year":2021,"id":8324,"datarank":6.520399318493976,"base_score":7.288244401020124,"endowment":7.288244401020124,"self_citation_contribution":1.0932366601530188,"citation_network_contribution":5.427162658340958,"self_endowment_contribution":1.0932366601530188,"citer_contribution":5.427162658340958,"corpus_percentile":97.08362342384157,"corpus_rank":378,"citation_count":1462,"citer_count":100,"citers_with_citation_signal":100,"citers_with_endowment":100,"datacite_reuse_total":0,"is_dataset":true,"is_dataset_confidence":0.8497,"is_data_producer":false,"deposit_databanks":null,"is_oa":true,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":"2021-12-23","fair_score":58.3333,"fair_percentile":72.8829104249465,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":3572,"name":"Maren Büttner","orcid":"0000-0002-6189-3792","position":1,"is_corresponding":false},{"id":11450,"name":"Kridsadakorn Chaichoompu","orcid":"0000-0002-8123-3409","position":2,"is_corresponding":false},{"id":73061,"name":"M. F. Mueller","orcid":null,"position":5,"is_corresponding":false},{"id":42,"name":"Fabian Joachim Theis","orcid":"0000-0002-2419-1943","position":10,"is_corresponding":false},{"id":11443,"name":"Anna Danese","orcid":"0000-0001-8810-4346","position":11,"is_corresponding":false},{"id":11444,"name":"Marta Interlandi","orcid":"0000-0002-6863-2552","position":12,"is_corresponding":false},{"id":73066,"name":"Martin Mueller","orcid":null,"position":13,"is_corresponding":false},{"id":11446,"name":"Daniel Strobl","orcid":"0000-0002-5516-7057","position":14,"is_corresponding":false},{"id":11447,"name":"Luke Zappia","orcid":"0000-0001-7744-8565","position":15,"is_corresponding":false},{"id":11448,"name":"Martin Dugas","orcid":"0000-0001-9740-0788","position":16,"is_corresponding":false},{"id":11449,"name":"Maria Colomé‐Tatché","orcid":"0000-0002-2224-7560","position":17,"is_corresponding":false},{"id":3578,"name":"Malte D. Luecken","orcid":"0000-0001-7464-7921","position":0,"is_corresponding":true}],"reference_count":52,"raw_metadata":{"citation_network_status":"fetched"},"created_at":"2026-03-01T18:20:47.508186Z","pmid":"34949812","pmcid":"PMC8748196","fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":"hybrid","license":"cc-by","views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":77.7778,"fair_a":50.0,"fair_i":20.0,"fair_r":50.0,"fair_zscore":0.9454,"fair_rationale":{"fair_score":58.33,"has_llm":true,"taxonomy_version":"fair_taxonomy_v5","dimensions":{"F":{"name":"Findable","score":77.78,"criteria":[{"key":"f_dataset_pid","label":"Persistent identifier for the data","kind":"llm","weight":2.0,"fraction":1.0,"verdict":"yes","evidence":"10.6084/m9.figshare.12420968","grounded":true,"rationale":"The paper provides a DOI for the dataset deposited on Figshare. [majority verdict 'yes' (4/5 passes agreed)]","anchors":["RDA-F1-01D — FAIR Data Maturity Model: 'Data is identified by a persistent identifier' (priorit","RDA-F1-02D — FAIR Data Maturity Model: 'Data is identified by a globally unique identifier'","FsF-F1-02D — F-UJI/FAIRsFAIR: 'Data is assigned a persistent identifier'"],"scored":true,"signal":null},{"key":"f_repository_named","label":"Named repository","kind":"llm","weight":2.0,"fraction":1.0,"verdict":"yes","evidence":"Figshare","grounded":true,"rationale":"Figshare is a recognised data repository named as the holder of the data. [majority verdict 'yes' (4/5 passes agreed)]","anchors":["RDA-F4-01M — FAIR Data Maturity Model: metadata is offered so it can be harvested and indexed (","NIH DMS Policy Element 4 (NOT-OD-21-014) — name the repository where data will be archived","NSTC Desirable Characteristics of Data Repositories (2022) — 'Long-Term Sustainability', 'Reten"],"scored":true,"signal":null},{"key":"f_data_availability_statement","label":"Data-availability statement","kind":"llm","weight":2.0,"fraction":0.5,"verdict":"partial","evidence":"Our reprocessed versions of these datasets are publicly available as preprocessed Anndata objects on Figshare (10.6084/m9.figshare.12420968). The output data from all metric runs are available in Supplementary Data 1.","grounded":false,"rationale":"The statement points to a repository record with a DOI (Figshare) and a supplementary file, matching Colavizza category 3. [downgraded to 'partial' — no verifiable quote from the paper]","anchors":["Colavizza, Hrynaszkiewicz, Staden, Whitaker & McGillivray (2020), 'The citation advantage of li","Springer Nature research data policy — Data Availability Statements: standard statement templat","RDA-F3-01M — metadata clearly and explicitly includes the identifier of the data it describes"],"scored":false,"signal":null},{"key":"f_discovery_metadata","label":"Description of the dataset as an object","kind":"llm","weight":2.0,"fraction":1.0,"verdict":"yes","evidence":"Table 1 Integration tasks for benchmarking","grounded":true,"rationale":"The paper includes an itemised table describing the datasets (cell numbers, batches, etc.). [majority verdict 'yes' (3/5 passes agreed)]","anchors":["RDA-F2-01M — 'Rich metadata is provided to allow discovery' (priority Essential)","FsF-F2-01M — F-UJI: 'Metadata includes descriptive core elements to support data findability'","FsF-R1-01MD — F-UJI: 'Metadata specifies the content of the data'"],"scored":false,"signal":null},{"key":"f_dataset_cited","label":"Dataset formally cited","kind":"llm","weight":1.0,"fraction":0.0,"verdict":"no","evidence":"Our reprocessed versions of these datasets are publicly available as preprocessed Anndata objects on Figshare (10.6084/m9.figshare.12420968).","grounded":false,"rationale":"The dataset identifier appears only in the body text (Data Availability section), not in the reference list. [downgraded to 'no' — no verifiable quote from the paper] [majority verdict 'no' (3/5 passes agreed)]","anchors":["FORCE11 Joint Declaration of Data Citation Principles (2014) — data should be cited as a first-","RDA-F3-01M — metadata clearly and explicitly includes the identifier of the data it describes","FsF-F3-01M — F-UJI: 'Metadata includes the identifier of the data it describes'"],"scored":true,"signal":null}]},"A":{"name":"Accessible","score":50.0,"criteria":[{"key":"a_data_openly_accessible","label":"Access route free of preconditions","kind":"llm","weight":2.0,"fraction":0.5,"verdict":"partial","evidence":"Our reprocessed versions of these datasets are publicly available as preprocessed Anndata objects on Figshare (10.6084/m9.figshare.12420968).","grounded":false,"rationale":"The data are stated to be publicly available on Figshare with no precondition. [downgraded to 'partial' — no verifiable quote from the paper]","anchors":["RDA-A1.1-01D — 'Data is accessible through a free access protocol'","FsF-A1-01M — F-UJI: 'Metadata contains access level and access conditions of the data'","NSTC Desirable Characteristics of Data Repositories (2022) — 'Free and Easy Access'"],"scored":true,"signal":null},{"key":"a_access_conditions_stated","label":"Access level labelled","kind":"llm","weight":1.0,"fraction":1.0,"verdict":"yes","evidence":"publicly available","grounded":true,"rationale":"The paper labels the data as 'publicly available', which is an access-level label equivalent to open access. [majority verdict 'yes' (3/5 passes agreed)]","anchors":["FsF-A1-01M — F-UJI: 'Metadata contains access level and access conditions of the data'","RDA-A1-01M — metadata contains information to enable the user to get access to the data","COAR Controlled Vocabularies — Access Rights v1.0 (open / embargoed / restricted / metadata-onl"],"scored":false,"signal":null},{"key":"a_controlled_access_for_sensitive","label":"Gatekeeper for sensitive data","kind":"llm","weight":0.5,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"The data are not sensitive and no gatekeeper is named; the data are publicly available on Figshare.","anchors":["NIH Genomic Data Sharing Policy (NOT-OD-14-124) — controlled-access via a Data Access Committee","RDA-A1.2-01D — 'Data is accessible through an access protocol that supports authentication and ","NIH DMS Policy Element 5 (NOT-OD-21-014) — Access, Distribution, or Reuse Considerations (conse"],"scored":false,"signal":null},{"key":"a_timeline_retention","label":"Availability timing & retention","kind":"llm","weight":0.5,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No sentence states how long the data will remain available or when they become obtainable beyond being publicly available.","anchors":["NIH DMS Plan Element 4 (NOT-OD-21-014) — Data Preservation, Access, and Associated Timelines","NSTC Desirable Characteristics (2022), Organizational Infrastructure: 'Retention Policy'","RDA-A2-01M — 'Metadata is guaranteed to remain available after data is no longer available'"],"scored":false,"signal":null}]},"I":{"name":"Interoperable","score":20.0,"criteria":[{"key":"i_open_nonproprietary_format","label":"Open file format","kind":"llm","weight":1.0,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"The paper does not name an open file format from the recognised list; 'Anndata objects' is not a standard format token. [majority verdict 'no' (4/5 passes agreed)]","anchors":["FsF-R1.3-02D — F-UJI: 'Data is available in a file format recommended by the target research co","RDA-R1.3-02D — data is expressed in a machine-understandable community standard","RDA-I1-01D — data uses a knowledge representation expressed in a standardised format"],"scored":true,"signal":null},{"key":"i_community_standard_vocabulary","label":"Community standard / vocabulary","kind":"llm","weight":1.0,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No data/metadata community standard (e.g., MIAME, MINSEQE, a controlled vocabulary) is named for the data.","anchors":["RDA-R1.3-01M — 'Metadata complies with a community standard' (priority Essential)","RDA-R1.3-01D — 'Data complies with a community standard'","RDA-I2-01M — '(Meta)data use vocabularies that follow FAIR principles'"],"scored":false,"signal":null},{"key":"i_qualified_references","label":"Identifiers for the resources the data depend on","kind":"llm","weight":0.5,"fraction":1.0,"verdict":"yes","evidence":"pancreas GSE81076 , GSE85241 , GSE86469 , GSE84133 , GSE81608 (Gene Expression Omnibus (GEO))","grounded":true,"rationale":"The paper provides identifiers (GEO accessions) for external source datasets used in the study.","anchors":["RDA-I3-01M — '(meta)data include references to other (meta)data'","RDA-I3-03M — 'metadata includes qualified references to other metadata'","FsF-I3-01M — F-UJI: 'Metadata includes links between the data and its related entities'"],"scored":false,"signal":null}]},"R":{"name":"Reusable","score":50.0,"criteria":[{"key":"r_reuse_license","label":"Reuse licence","kind":"llm","weight":2.0,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No license for the data is stated; the article's CC BY license does not apply to the data.","anchors":["RDA-R1.1-01M — 'Metadata includes information about the licence under which the data can be reu","RDA-R1.1-02M — 'Metadata refers to a standard reuse licence'","RDA-R1.1-03M — 'Metadata refers to a machine-understandable reuse licence'"],"scored":true,"signal":null},{"key":"r_provenance_methods","label":"Provenance of the data","kind":"llm","weight":1.0,"fraction":1.0,"verdict":"yes","evidence":"We used scran pooling normalization 40 (v.1.10.2 unless otherwise specified) and log +1 transformation on count data.","grounded":true,"rationale":"The paper names specific tools (scran) and versions used to produce the data. [majority verdict 'yes' (3/5 passes agreed)]","anchors":["RDA-R1.2-01M — 'Metadata includes provenance information according to community- specific standa","FsF-R1.2-01M — F-UJI: 'Metadata includes provenance information about data creation or generati","W3C PROV-O (W3C Recommendation, 2013) — the entity/activity/agent model of provenance"],"scored":false,"signal":null},{"key":"r_documentation_codebook","label":"Documentation / codebook","kind":"llm","weight":1.0,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No documentation object (README, data dictionary, codebook) is named as accompanying the data. [majority verdict 'no' (4/5 passes agreed)]","anchors":["RDA-R1-01M — '(Meta)data are richly described with a plurality of accurate and relevant attribu","FsF-R1-01MD — F-UJI: 'Metadata specifies the content of the data'","NIH DMS Policy Element 3 (NOT-OD-21-014) — Standards (documentation and metadata to accompany t"],"scored":false,"signal":null},{"key":"r_versioning","label":"Snapshot identified","kind":"llm","weight":0.5,"fraction":1.0,"verdict":"yes","evidence":"10.6084/m9.figshare.12420968.v7","grounded":true,"rationale":"The Figshare DOI includes a version suffix (.v7), identifying the snapshot. [majority verdict 'yes' (3/5 passes agreed)]","anchors":["DataCite Metadata Schema 4.6 — the 'Version' property","RDA-R1.2-01M — provenance information (which version was used is provenance)","NSTC Desirable Characteristics of Data Repositories (2022) — 'Provenance', 'Retention Policy'"],"scored":true,"signal":null},{"key":"x_code_availability","label":"Analysis code available","kind":"llm","weight":1.0,"fraction":1.0,"verdict":"yes","evidence":"The preprocessing, integration and evaluation metric functions with relevant parameterizations have been made available is our scIB Python package at https://github.com/theislab/scib","grounded":true,"rationale":"A machine-resolvable code repository URL is given for the study's own code. [majority verdict 'yes' (4/5 passes agreed)]","anchors":["NIH DMS Policy Element 2 (NOT-OD-21-014) — 'Related Tools, Software and/or Code'","FAIR4RS Principles v1.0 (Chue Hong et al., 2022; RDA/FORCE11/ReSA) — FAIR Principles for Resear","FORCE11 Software Citation Principles (Smith, Katz & Niemeyer, 2016, PeerJ CS 2:e86)"],"scored":true,"signal":null},{"key":"x_funding_attribution","label":"Funder and award number","kind":"llm","weight":0.5,"fraction":1.0,"verdict":"yes","evidence":"ExNet-0041-Phase2-3","grounded":true,"rationale":"A specific award number is provided for a named funder.","anchors":["DataCite Metadata Schema 4.6 — 'FundingReference' property (funderName, funderIdentifier, award","Crossref Funder Registry — canonical funder identifiers for funding metadata","RDA-F2-01M — rich metadata provided to allow discovery (funding is part of the descriptive reco"],"scored":true,"signal":null}]}},"actions":[{"key":"r_reuse_license","dimension":"R","label":"Reuse licence","action":"Attach a standard, machine-readable open licence to the deposit — CC0 or CC BY, which is what Horizon Europe and most funders expect — and print the licence identifier in the paper. 'Free to use' is not a licence: it grants nothing a reuser's institution can rely on.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No license for the data is stated; the article's CC BY license does not apply to the data.","gain":16.67,"priority":"essential","scored":true},{"key":"a_data_openly_accessible","dimension":"A","label":"Access route free of preconditions","action":"Remove the precondition or justify it. Release the data at publication with no embargo, no registration wall, and no approval step — NIH's zero-embargo public- access rule (NOT-OD-25-101) has already made 'available at publication' the federal baseline for the article; the data should not lag behind it. For genomics / sequencing data, deposit in GEO (GSE accession), SRA (SRP/SRR) or ENA/BioProject (PRJEB/PRJNA).","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"Our reprocessed versions of these datasets are publicly available as preprocessed Anndata objects on Figshare (10.6084/m9.figshare.12420968).","why":"The data are stated to be publicly available on Figshare with no precondition. [downgraded to 'partial' — no verifiable quote from the paper]","gain":8.33,"priority":"essential","scored":true},{"key":"f_dataset_cited","dimension":"F","label":"Dataset formally cited","action":"Cite the dataset in the reference list like a publication — creator, year, title, repository, DOI/accession — and cite it in-text where it is used. Only a reference- list entry is machine-readable to Crossref/DataCite, and only a citation lets the data earn credit. Cite the genomics / sequencing repository accession (e.g. from GEO (GSE accession), SRA (SRP/SRR) or ENA/BioProject (PRJEB/PRJNA)) in the reference list.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":"Our reprocessed versions of these datasets are publicly available as preprocessed Anndata objects on Figshare (10.6084/m9.figshare.12420968).","why":"The dataset identifier appears only in the body text (Data Availability section), not in the reference list. [downgraded to 'no' — no verifiable quote from the paper] [majority verdict 'no' (3/5 passes agreed)]","gain":8.33,"priority":"important","scored":true},{"key":"i_open_nonproprietary_format","dimension":"I","label":"Open file format","action":"Release the data in an open, community-standard format (CSV/TSV, JSON, HDF5, NetCDF, FASTQ, VCF, NIfTI…) instead of — or alongside — any proprietary or instrument-native format, and name the format in the paper. A dataset that needs a €2,000 licence to open is not reusable. Prefer open genomics / sequencing formats such as FASTQ, BAM or VCF.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"The paper does not name an open file format from the recognised list; 'Anndata objects' is not a standard format token. [majority verdict 'no' (4/5 passes agreed)]","gain":8.33,"priority":"important","scored":true},{"key":"f_data_availability_statement","dimension":"F","label":"Data-availability statement","action":"Replace the statement with the repository template: name the repository and give the accession or DOI (Colavizza category 3). This is the only DAS class associated with a measured citation advantage; 'available on reasonable request' and 'within the article' are not.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"Our reprocessed versions of these datasets are publicly available as preprocessed Anndata objects on Figshare (10.6084/m9.figshare.12420968). The output data from all metric runs are available in Supplementary Data 1.","why":"The statement points to a repository record with a DOI (Figshare) and a supplementary file, matching Colavizza category 3. [downgraded to 'partial' — no verifiable quote from the paper]","gain":0.0,"priority":"essential","scored":false},{"key":"i_community_standard_vocabulary","dimension":"I","label":"Community standard / vocabulary","action":"Adopt and NAME your domain's data standard — the minimum-information checklist, metadata schema, or ontology your community uses (MIAME/MINSEQE, ISA-Tab, BIDS, an OBO ontology, HL7 FHIR/OMOP) — and say which one you followed. A reporting checklist standardises your paper; it does nothing for your data. In genomics / sequencing, describe the data with MIAME, MINSEQE or MIxS.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No data/metadata community standard (e.g., MIAME, MINSEQE, a controlled vocabulary) is named for the data.","gain":0.0,"priority":"important","scored":false},{"key":"r_documentation_codebook","dimension":"R","label":"Documentation / codebook","action":"Ship a README and a data dictionary IN the deposit — every file, every variable, its units, its allowed values, its missing-value codes. It is the cheapest single thing that makes a dataset usable by someone who was not in the lab, and a table buried in the article does not travel with the data.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No documentation object (README, data dictionary, codebook) is named as accompanying the data. [majority verdict 'no' (4/5 passes agreed)]","gain":0.0,"priority":"important","scored":false},{"key":"a_controlled_access_for_sensitive","dimension":"A","label":"Gatekeeper for sensitive data","action":"Route sensitive data through an institutional gatekeeper — deposit in a controlled- access repository (dbGaP, EGA) with a Data Access Committee and a published DUA — rather than through the corresponding author's inbox. An author-gated dataset dies with the author's email address, and 'on reasonable request' has been shown repeatedly not to yield data. For sensitive/human genomics / sequencing data, use a controlled-access repository such as dbGaP or EGA.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"The data are not sensitive and no gatekeeper is named; the data are publicly available on Figshare.","gain":0.0,"priority":"useful","scored":false},{"key":"a_timeline_retention","dimension":"A","label":"Availability timing & retention","action":"State when the data become available AND how long they will be retained — cite the repository's preservation policy. NIH DMS Element 4 asks for both; most papers give neither.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No sentence states how long the data will remain available or when they become obtainable beyond being publicly available.","gain":0.0,"priority":"useful","scored":false}],"suggestions":["Attach a standard, machine-readable open licence to the deposit — CC0 or CC BY, which is what Horizon Europe and most funders expect — and print the licence identifier in the paper. 'Free to use' is not a licence: it grants nothing a reuser's institution can rely on.","Remove the precondition or justify it. Release the data at publication with no embargo, no registration wall, and no approval step — NIH's zero-embargo public- access rule (NOT-OD-25-101) has already made 'available at publication' the federal baseline for the article; the data should not lag behind it. For genomics / sequencing data, deposit in GEO (GSE accession), SRA (SRP/SRR) or ENA/BioProject (PRJEB/PRJNA).","Cite the dataset in the reference list like a publication — creator, year, title, repository, DOI/accession — and cite it in-text where it is used. Only a reference- list entry is machine-readable to Crossref/DataCite, and only a citation lets the data earn credit. Cite the genomics / sequencing repository accession (e.g. from GEO (GSE accession), SRA (SRP/SRR) or ENA/BioProject (PRJEB/PRJNA)) in the reference list.","Release the data in an open, community-standard format (CSV/TSV, JSON, HDF5, NetCDF, FASTQ, VCF, NIfTI…) instead of — or alongside — any proprietary or instrument-native format, and name the format in the paper. A dataset that needs a €2,000 licence to open is not reusable. Prefer open genomics / sequencing formats such as FASTQ, BAM or VCF.","Replace the statement with the repository template: name the repository and give the accession or DOI (Colavizza category 3). This is the only DAS class associated with a measured citation advantage; 'available on reasonable request' and 'within the article' are not."],"model":"deepseek/deepseek-v4-flash","agent_version":"fair_agent_v8","fulltext_source":"epmc_xml"},"fair_model":"deepseek/deepseek-v4-flash","fair_agent_version":"fair_agent_v8","fair_fulltext_source":"epmc_xml","fair_has_llm":true,"fair_computed_at":"2026-07-20T10:45:50.690332Z","clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}