{"doi":"10.3389/frma.2021.674205","title":"ECO-CollecTF: A Corpus of Annotated Evidence-Based Assertions in Biomedical Manuscripts","abstract":"Analysis of high-throughput experiments in the life sciences frequently relies upon standardized information about genes, gene products, and other biological entities. To provide this information, expert curators are increasingly relying on text mining tools to identify, extract and harmonize statements from biomedical journal articles that discuss findings of interest. For determining reliability of the statements, curators need the evidence used by the authors to support their assertions. It is important to annotate the evidence directly used by authors to qualify their findings rather than simply annotating mentions of experimental methods without the context of what findings they support. Text mining tools require tuning and adaptation to achieve accurate performance. Many annotated corpora exist to enable developing and tuning text mining tools; however, none currently provides annotations of evidence based on the extensive and widely used Evidence and Conclusion Ontology. We present the ECO-CollecTF corpus, a novel, freely available, biomedical corpus of 84 documents that captures high-quality, evidence-based statements annotated with the Evidence and Conclusion Ontology.","journal":"Frontiers in Research Metrics and Analytics","year":2021,"id":210703,"datarank":0.24582358959035067,"base_score":1.3862943611198906,"endowment":1.3862943611198906,"self_citation_contribution":0.20794415416798362,"citation_network_contribution":0.03787943542236703,"self_endowment_contribution":0.20794415416798362,"citer_contribution":0.03787943542236703,"corpus_percentile":39.900982439854566,"corpus_rank":7770,"citation_count":3,"citer_count":2,"citers_with_citation_signal":2,"citers_with_endowment":2,"datacite_reuse_total":0,"is_dataset":true,"is_dataset_confidence":0.7678,"is_data_producer":false,"deposit_databanks":null,"is_oa":true,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":"2021-01-01","fair_score":81.25,"fair_percentile":98.68541730357688,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":679374,"name":"Stephen M. Goralski","orcid":null,"position":1,"is_corresponding":false},{"id":799655,"name":"Ashley Mitchell","orcid":null,"position":2,"is_corresponding":false},{"id":799221,"name":"Andrew Simpson","orcid":"0000-0001-6584-6451","position":3,"is_corresponding":false},{"id":799656,"name":"Dorjan Leka","orcid":null,"position":4,"is_corresponding":false},{"id":799657,"name":"Emmanuel Kotey","orcid":null,"position":5,"is_corresponding":false},{"id":799658,"name":"Matt Sekira","orcid":null,"position":6,"is_corresponding":false},{"id":70332,"name":"James B. Munro","orcid":"0000-0003-3067-6434","position":7,"is_corresponding":false},{"id":70333,"name":"Suvarna Nadendla","orcid":"0000-0003-3643-281X","position":8,"is_corresponding":false},{"id":103836,"name":"Rebecca Jackson","orcid":"0000-0003-4871-5569","position":9,"is_corresponding":false},{"id":799659,"name":"Aitor Gonzalez-Aguirre","orcid":null,"position":10,"is_corresponding":false},{"id":531515,"name":"Martin Krallinger","orcid":"0000-0002-2646-8782","position":11,"is_corresponding":false},{"id":29022,"name":"Michelle Giglio","orcid":"0000-0001-7628-5565","position":12,"is_corresponding":false},{"id":678460,"name":"Ivan Erill","orcid":"0000-0002-7280-7191","position":13,"is_corresponding":false},{"id":679373,"name":"Elizabeth T. Hobbs","orcid":null,"position":0,"is_corresponding":true}],"reference_count":62,"raw_metadata":null,"created_at":"2026-07-18T23:52:12.491146Z","pmid":"34327299","pmcid":"PMC8313968","fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":null,"license":null,"views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":94.4444,"fair_a":87.5,"fair_i":80.0,"fair_r":70.8333,"fair_zscore":1.8524,"fair_rationale":{"fair_score":81.25,"has_llm":true,"taxonomy_version":"fair_taxonomy_v5","dimensions":{"F":{"name":"Findable","score":94.44,"criteria":[{"key":"f_dataset_pid","label":"Persistent identifier for the data","kind":"llm","weight":2.0,"fraction":1.0,"verdict":"yes","evidence":"DOI: 10.5281/zenodo.4568935","grounded":true,"rationale":"The paper provides a DOI for the dataset, which is a persistent identifier scheme. [majority verdict 'yes' (4/5 passes agreed)]","anchors":["RDA-F1-01D — FAIR Data Maturity Model: 'Data is identified by a persistent identifier' (priorit","RDA-F1-02D — FAIR Data Maturity Model: 'Data is identified by a globally unique identifier'","FsF-F1-02D — F-UJI/FAIRsFAIR: 'Data is assigned a persistent identifier'"],"scored":true,"signal":null},{"key":"f_repository_named","label":"Named repository","kind":"llm","weight":2.0,"fraction":1.0,"verdict":"yes","evidence":"deposited in Zenodo for persistence","grounded":true,"rationale":"Zenodo is a named data repository. [majority verdict 'yes' (4/5 passes agreed)]","anchors":["RDA-F4-01M — FAIR Data Maturity Model: metadata is offered so it can be harvested and indexed (","NIH DMS Policy Element 4 (NOT-OD-21-014) — name the repository where data will be archived","NSTC Desirable Characteristics of Data Repositories (2022) — 'Long-Term Sustainability', 'Reten"],"scored":true,"signal":null},{"key":"f_data_availability_statement","label":"Data-availability statement","kind":"llm","weight":2.0,"fraction":1.0,"verdict":"yes","evidence":"The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found below: ECO-CollecTF v1.2 https://zenodo.org/record/4568935 . DOI: 10.5281/zenodo.4568935 .","grounded":true,"rationale":"The statement points to a repository record with a DOI (Colavizza category 3). [majority verdict 'yes' (4/5 passes agreed)]","anchors":["Colavizza, Hrynaszkiewicz, Staden, Whitaker & McGillivray (2020), 'The citation advantage of li","Springer Nature research data policy — Data Availability Statements: standard statement templat","RDA-F3-01M — metadata clearly and explicitly includes the identifier of the data it describes"],"scored":false,"signal":null},{"key":"f_discovery_metadata","label":"Description of the dataset as an object","kind":"llm","weight":2.0,"fraction":1.0,"verdict":"yes","evidence":"TABLE 2 ECO-CollecTF corpus statistics.","grounded":true,"rationale":"The paper includes a table with itemised statistics of the dataset. [majority verdict 'yes' (3/5 passes agreed)]","anchors":["RDA-F2-01M — 'Rich metadata is provided to allow discovery' (priority Essential)","FsF-F2-01M — F-UJI: 'Metadata includes descriptive core elements to support data findability'","FsF-R1-01MD — F-UJI: 'Metadata specifies the content of the data'"],"scored":false,"signal":null},{"key":"f_dataset_cited","label":"Dataset formally cited","kind":"llm","weight":1.0,"fraction":0.5,"verdict":"partial","evidence":"ECO-CollecTF v1.2 https://zenodo.org/record/4568935 . DOI: 10.5281/zenodo.4568935 .","grounded":true,"rationale":"The dataset identifier appears only in the body text of the Data Availability Statement, not as a reference-list entry. [majority verdict 'partial' (3/5 passes agreed)]","anchors":["FORCE11 Joint Declaration of Data Citation Principles (2014) — data should be cited as a first-","RDA-F3-01M — metadata clearly and explicitly includes the identifier of the data it describes","FsF-F3-01M — F-UJI: 'Metadata includes the identifier of the data it describes'"],"scored":true,"signal":null}]},"A":{"name":"Accessible","score":87.5,"criteria":[{"key":"a_data_openly_accessible","label":"Access route free of preconditions","kind":"llm","weight":2.0,"fraction":1.0,"verdict":"yes","evidence":"A permanent, open access and freely available version of the corpus is accessible via the ECO website ( http://evidenceontology.org/annotation_resources/ ) and deposited in Zenodo for persistence (DOI: 10.5281/zenodo.4568935 ).","grounded":true,"rationale":"The text states the data are open access and freely available with no precondition. [majority verdict 'yes' (4/5 passes agreed)]","anchors":["RDA-A1.1-01D — 'Data is accessible through a free access protocol'","FsF-A1-01M — F-UJI: 'Metadata contains access level and access conditions of the data'","NSTC Desirable Characteristics of Data Repositories (2022) — 'Free and Easy Access'"],"scored":true,"signal":null},{"key":"a_access_conditions_stated","label":"Access level labelled","kind":"llm","weight":1.0,"fraction":1.0,"verdict":"yes","evidence":"A permanent, open access and freely available version of the corpus is accessible via the ECO website ( http://evidenceontology.org/annotation_resources/ ) and deposited in Zenodo for persistence (DOI: 10.5281/zenodo.4568935 ).","grounded":true,"rationale":"The paper explicitly labels the data as 'open access' and 'freely available'.","anchors":["FsF-A1-01M — F-UJI: 'Metadata contains access level and access conditions of the data'","RDA-A1-01M — metadata contains information to enable the user to get access to the data","COAR Controlled Vocabularies — Access Rights v1.0 (open / embargoed / restricted / metadata-onl"],"scored":false,"signal":null},{"key":"a_controlled_access_for_sensitive","label":"Gatekeeper for sensitive data","kind":"llm","weight":0.5,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"The data are not sensitive human-subject data, and no gatekeeper is named; the data are openly accessible without restriction.","anchors":["NIH Genomic Data Sharing Policy (NOT-OD-14-124) — controlled-access via a Data Access Committee","RDA-A1.2-01D — 'Data is accessible through an access protocol that supports authentication and ","NIH DMS Policy Element 5 (NOT-OD-21-014) — Access, Distribution, or Reuse Considerations (conse"],"scored":false,"signal":null},{"key":"a_timeline_retention","label":"Availability timing & retention","kind":"llm","weight":0.5,"fraction":1.0,"verdict":"yes","evidence":"A permanent, open access and freely available version of the corpus is accessible via the ECO website ( http://evidenceontology.org/annotation_resources/ ) and deposited in Zenodo for persistence (DOI: 10.5281/zenodo.4568935 ).","grounded":true,"rationale":"The word 'permanent' and the deposit in Zenodo indicate a persistence commitment.","anchors":["NIH DMS Plan Element 4 (NOT-OD-21-014) — Data Preservation, Access, and Associated Timelines","NSTC Desirable Characteristics (2022), Organizational Infrastructure: 'Retention Policy'","RDA-A2-01M — 'Metadata is guaranteed to remain available after data is no longer available'"],"scored":false,"signal":null}]},"I":{"name":"Interoperable","score":80.0,"criteria":[{"key":"i_open_nonproprietary_format","label":"Open file format","kind":"llm","weight":1.0,"fraction":0.5,"verdict":"partial","evidence":"available in BRAT (Stenetorp et al., 2012), a de facto standard format for biomedical corpora, as well as in BioC (Comeau et al., 2013)","grounded":false,"rationale":"Both BRAT and BioC are open, community-standard formats. [downgraded to 'partial' — no verifiable quote from the paper] [majority verdict 'partial' (3/5 passes agreed)]","anchors":["FsF-R1.3-02D — F-UJI: 'Data is available in a file format recommended by the target research co","RDA-R1.3-02D — data is expressed in a machine-understandable community standard","RDA-I1-01D — data uses a knowledge representation expressed in a standardised format"],"scored":true,"signal":null},{"key":"i_community_standard_vocabulary","label":"Community standard / vocabulary","kind":"llm","weight":1.0,"fraction":1.0,"verdict":"yes","evidence":"captured using the Evidence and Conclusion Ontology (ECO)","grounded":true,"rationale":"ECO is a community-standard ontology registered in FAIRsharing.","anchors":["RDA-R1.3-01M — 'Metadata complies with a community standard' (priority Essential)","RDA-R1.3-01D — 'Data complies with a community standard'","RDA-I2-01M — '(Meta)data use vocabularies that follow FAIR principles'"],"scored":false,"signal":null},{"key":"i_qualified_references","label":"Identifiers for the resources the data depend on","kind":"llm","weight":0.5,"fraction":1.0,"verdict":"yes","evidence":"https://github.com/ErillLab/ECO-CollecTF/tree/master/code/PrepFiles","grounded":true,"rationale":"The paper provides a GitHub URL for the code used to prepare the texts. [majority verdict 'yes' (3/5 passes agreed)]","anchors":["RDA-I3-01M — '(meta)data include references to other (meta)data'","RDA-I3-03M — 'metadata includes qualified references to other metadata'","FsF-I3-01M — F-UJI: 'Metadata includes links between the data and its related entities'"],"scored":false,"signal":null}]},"R":{"name":"Reusable","score":70.83,"criteria":[{"key":"r_reuse_license","label":"Reuse licence","kind":"llm","weight":2.0,"fraction":0.5,"verdict":"partial","evidence":"Annotations are available according to Creative Commons BY NC 4.0 license.","grounded":true,"rationale":"CC BY-NC 4.0 is a named license but not on the open list of standard open licenses.","anchors":["RDA-R1.1-01M — 'Metadata includes information about the licence under which the data can be reu","RDA-R1.1-02M — 'Metadata refers to a standard reuse licence'","RDA-R1.1-03M — 'Metadata refers to a machine-understandable reuse licence'"],"scored":true,"signal":null},{"key":"r_provenance_methods","label":"Provenance of the data","kind":"llm","weight":1.0,"fraction":1.0,"verdict":"yes","evidence":"We used the current version of BRAT, v1.3 “Crunchy Frog”.","grounded":true,"rationale":"The paper names specific tools (BRAT version) used to produce the data. [majority verdict 'yes' (3/5 passes agreed)]","anchors":["RDA-R1.2-01M — 'Metadata includes provenance information according to community- specific standa","FsF-R1.2-01M — F-UJI: 'Metadata includes provenance information about data creation or generati","W3C PROV-O (W3C Recommendation, 2013) — the entity/activity/agent model of provenance"],"scored":false,"signal":null},{"key":"r_documentation_codebook","label":"Documentation / codebook","kind":"llm","weight":1.0,"fraction":0.5,"verdict":"partial","evidence":"This section describes the annotation guidelines ( Supplementary Material 1 ).","grounded":true,"rationale":"The variable definitions are provided inside the article (Supplementary Material), not as a separate file shipped with the data. [majority verdict 'partial' (3/5 passes agreed)]","anchors":["RDA-R1-01M — '(Meta)data are richly described with a plurality of accurate and relevant attribu","FsF-R1-01MD — F-UJI: 'Metadata specifies the content of the data'","NIH DMS Policy Element 3 (NOT-OD-21-014) — Standards (documentation and metadata to accompany t"],"scored":false,"signal":null},{"key":"r_versioning","label":"Snapshot identified","kind":"llm","weight":0.5,"fraction":1.0,"verdict":"yes","evidence":"ECO-CollecTF v1.2","grounded":true,"rationale":"The paper provides a version token for the dataset.","anchors":["DataCite Metadata Schema 4.6 — the 'Version' property","RDA-R1.2-01M — provenance information (which version was used is provenance)","NSTC Desirable Characteristics of Data Repositories (2022) — 'Provenance', 'Retention Policy'"],"scored":true,"signal":null},{"key":"x_code_availability","label":"Analysis code available","kind":"llm","weight":1.0,"fraction":1.0,"verdict":"yes","evidence":"All scripts used to prepare the texts and instructions are available at https://github.com/ErillLab/ECO-CollecTF/tree/master/code/PrepFiles .","grounded":true,"rationale":"A machine-resolvable URL to the code repository is given.","anchors":["NIH DMS Policy Element 2 (NOT-OD-21-014) — 'Related Tools, Software and/or Code'","FAIR4RS Principles v1.0 (Chue Hong et al., 2022; RDA/FORCE11/ReSA) — FAIR Principles for Resear","FORCE11 Software Citation Principles (Smith, Katz & Niemeyer, 2016, PeerJ CS 2:e86)"],"scored":true,"signal":null},{"key":"x_funding_attribution","label":"Funder and award number","kind":"llm","weight":0.5,"fraction":0.5,"verdict":"partial","evidence":"This work was supported by the National Science Foundation, Division of Biological Infrastructure (1458400) and the National Institutes of Health (R01GM089636, U41HG008735).","grounded":false,"rationale":"Award numbers are provided for the funding. [downgraded to 'partial' — no verifiable quote from the paper] [majority verdict 'partial' (3/5 passes agreed)]","anchors":["DataCite Metadata Schema 4.6 — 'FundingReference' property (funderName, funderIdentifier, award","Crossref Funder Registry — canonical funder identifiers for funding metadata","RDA-F2-01M — rich metadata provided to allow discovery (funding is part of the descriptive reco"],"scored":true,"signal":null}]}},"actions":[{"key":"r_reuse_license","dimension":"R","label":"Reuse licence","action":"Attach a standard, machine-readable open licence to the deposit — CC0 or CC BY, which is what Horizon Europe and most funders expect — and print the licence identifier in the paper. 'Free to use' is not a licence: it grants nothing a reuser's institution can rely on.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"Annotations are available according to Creative Commons BY NC 4.0 license.","why":"CC BY-NC 4.0 is a named license but not on the open list of standard open licenses.","gain":8.33,"priority":"essential","scored":true},{"key":"f_dataset_cited","dimension":"F","label":"Dataset formally cited","action":"Cite the dataset in the reference list like a publication — creator, year, title, repository, DOI/accession — and cite it in-text where it is used. Only a reference- list entry is machine-readable to Crossref/DataCite, and only a citation lets the data earn credit. Cite the genomics / sequencing repository accession (e.g. from GEO (GSE accession), SRA (SRP/SRR) or ENA/BioProject (PRJEB/PRJNA)) in the reference list.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"ECO-CollecTF v1.2 https://zenodo.org/record/4568935 . DOI: 10.5281/zenodo.4568935 .","why":"The dataset identifier appears only in the body text of the Data Availability Statement, not as a reference-list entry. [majority verdict 'partial' (3/5 passes agreed)]","gain":4.17,"priority":"important","scored":true},{"key":"i_open_nonproprietary_format","dimension":"I","label":"Open file format","action":"Release the data in an open, community-standard format (CSV/TSV, JSON, HDF5, NetCDF, FASTQ, VCF, NIfTI…) instead of — or alongside — any proprietary or instrument-native format, and name the format in the paper. A dataset that needs a €2,000 licence to open is not reusable. Prefer open genomics / sequencing formats such as FASTQ, BAM or VCF.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"available in BRAT (Stenetorp et al., 2012), a de facto standard format for biomedical corpora, as well as in BioC (Comeau et al., 2013)","why":"Both BRAT and BioC are open, community-standard formats. [downgraded to 'partial' — no verifiable quote from the paper] [majority verdict 'partial' (3/5 passes agreed)]","gain":4.17,"priority":"important","scored":true},{"key":"x_funding_attribution","dimension":"R","label":"Funder and award number","action":"State the funder AND the award number in the paper, and put them in the dataset's FundingReference metadata. A funder name alone cannot be linked back to the award, so the funding provenance of the data is lost the moment the paper is indexed.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"This work was supported by the National Science Foundation, Division of Biological Infrastructure (1458400) and the National Institutes of Health (R01GM089636, U41HG008735).","why":"Award numbers are provided for the funding. [downgraded to 'partial' — no verifiable quote from the paper] [majority verdict 'partial' (3/5 passes agreed)]","gain":2.08,"priority":"useful","scored":true},{"key":"r_documentation_codebook","dimension":"R","label":"Documentation / codebook","action":"Ship a README and a data dictionary IN the deposit — every file, every variable, its units, its allowed values, its missing-value codes. It is the cheapest single thing that makes a dataset usable by someone who was not in the lab, and a table buried in the article does not travel with the data.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"This section describes the annotation guidelines ( Supplementary Material 1 ).","why":"The variable definitions are provided inside the article (Supplementary Material), not as a separate file shipped with the data. [majority verdict 'partial' (3/5 passes agreed)]","gain":0.0,"priority":"important","scored":false},{"key":"a_controlled_access_for_sensitive","dimension":"A","label":"Gatekeeper for sensitive data","action":"Route sensitive data through an institutional gatekeeper — deposit in a controlled- access repository (dbGaP, EGA) with a Data Access Committee and a published DUA — rather than through the corresponding author's inbox. An author-gated dataset dies with the author's email address, and 'on reasonable request' has been shown repeatedly not to yield data. For sensitive/human genomics / sequencing data, use a controlled-access repository such as dbGaP or EGA.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"The data are not sensitive human-subject data, and no gatekeeper is named; the data are openly accessible without restriction.","gain":0.0,"priority":"useful","scored":false}],"suggestions":["Attach a standard, machine-readable open licence to the deposit — CC0 or CC BY, which is what Horizon Europe and most funders expect — and print the licence identifier in the paper. 'Free to use' is not a licence: it grants nothing a reuser's institution can rely on.","Cite the dataset in the reference list like a publication — creator, year, title, repository, DOI/accession — and cite it in-text where it is used. Only a reference- list entry is machine-readable to Crossref/DataCite, and only a citation lets the data earn credit. Cite the genomics / sequencing repository accession (e.g. from GEO (GSE accession), SRA (SRP/SRR) or ENA/BioProject (PRJEB/PRJNA)) in the reference list.","Release the data in an open, community-standard format (CSV/TSV, JSON, HDF5, NetCDF, FASTQ, VCF, NIfTI…) instead of — or alongside — any proprietary or instrument-native format, and name the format in the paper. A dataset that needs a €2,000 licence to open is not reusable. Prefer open genomics / sequencing formats such as FASTQ, BAM or VCF.","State the funder AND the award number in the paper, and put them in the dataset's FundingReference metadata. A funder name alone cannot be linked back to the award, so the funding provenance of the data is lost the moment the paper is indexed.","Ship a README and a data dictionary IN the deposit — every file, every variable, its units, its allowed values, its missing-value codes. It is the cheapest single thing that makes a dataset usable by someone who was not in the lab, and a table buried in the article does not travel with the data."],"model":"deepseek/deepseek-v4-flash","agent_version":"fair_agent_v8","fulltext_source":"epmc_xml"},"fair_model":"deepseek/deepseek-v4-flash","fair_agent_version":"fair_agent_v8","fair_fulltext_source":"epmc_xml","fair_has_llm":true,"fair_computed_at":"2026-07-20T13:17:08.132958Z","clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}