{"doi":"10.1371/journal.pcbi.1013239","title":"Verification and reproducible curation of the BioModels repository","abstract":"The BioModels Repository contains over 1000 manually curated mechanistic models from published literature, most often encoded in the Systems Biology Markup Language (SBML). This community-based standard formally specifies each model, but does not describe the computational experimental conditions to run a simulation and collect data. Therefore, it can be challenging to reproduce any figure or result from a publication with an SBML model alone. The Simulation Experiment Description Markup Language (SED-ML) provides a solution: a standard way to specify exactly how to run an experiment corresponding to a specific figure or result. BioModels was established years before SED-ML, and both systems evolved over time, both in content and acceptance. Hence, only about half of the entries in BioModels contained SED-ML files, and these files reflected the version of SED-ML that was available at the time. Additionally, almost all of these SED-ML files had at least one minor mistake that made them impossible to run. To make these models and their results more reproducible, we report here on our work updating, correcting and generating new SED-ML files for 1055 curated mechanistic models in BioModels. In addition, because SED-ML is implementation-independent, it can be used for verification, demonstrating that results hold across multiple simulation engines. We tested, corrected, and improved over 450 existing SED-ML files in the BioModels database, and created basic files for the rest of the entries. Then, we used a wrapper architecture for interpreting SED-ML, and report verification results across five different ODE-based biosimulation engines, after further improving the models, the wrappers, and the engines themselves. Our work with SED-ML and the BioModels collection aims to improve the utility of these models by making them more reproducible and credible. Improved reproducibility means these models are now even more fit for re-use, such as in new investigations and as components of multiscale models.","journal":"PLoS Computational Biology","year":2025,"id":584209,"datarank":0.0,"base_score":0.0,"endowment":0.0,"self_citation_contribution":0.0,"citation_network_contribution":0.0,"self_endowment_contribution":0.0,"citer_contribution":0.0,"corpus_percentile":0.0,"corpus_rank":10062,"citation_count":0,"citer_count":0,"citers_with_citation_signal":0,"citers_with_endowment":0,"datacite_reuse_total":0,"is_dataset":true,"is_dataset_confidence":0.9348,"is_data_producer":false,"deposit_databanks":null,"is_oa":true,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":"2025-01-01","fair_score":68.75,"fair_percentile":91.13420972179762,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":60853,"name":"Rahuman S. Malik‐Sheriff","orcid":"0000-0003-0705-9809","position":1,"is_corresponding":false},{"id":60860,"name":"Tung V. N. Nguyen","orcid":"0000-0002-2876-6046","position":2,"is_corresponding":false},{"id":5922,"name":"Henning Hermjakob","orcid":"0000-0001-8479-0262","position":3,"is_corresponding":false},{"id":60885,"name":"Jonathan R Karr","orcid":"0000-0002-2605-5080","position":4,"is_corresponding":false},{"id":60820,"name":"Bilal Shaikh","orcid":"0000-0001-5801-5510","position":5,"is_corresponding":false},{"id":1463043,"name":"Logan Drescher","orcid":null,"position":6,"is_corresponding":false},{"id":60884,"name":"Ion I Moraru","orcid":"0000-0002-3746-9676","position":7,"is_corresponding":false},{"id":60881,"name":"James C Schaff","orcid":"0000-0003-3286-7736","position":8,"is_corresponding":false},{"id":60825,"name":"Eran Agmon","orcid":"0000-0003-1279-2474","position":9,"is_corresponding":false},{"id":1463044,"name":"Alexander A. Patrie","orcid":null,"position":10,"is_corresponding":false},{"id":60882,"name":"Michael L Blinov","orcid":"0000-0002-9363-9705","position":11,"is_corresponding":false},{"id":60845,"name":"Joseph L. Hellerstein","orcid":"0000-0003-0802-4069","position":12,"is_corresponding":false},{"id":612541,"name":"Elebeoba E. May","orcid":null,"position":13,"is_corresponding":false},{"id":60861,"name":"David P Nickerson","orcid":"0000-0003-4667-9779","position":14,"is_corresponding":false},{"id":12964,"name":"John H. Gennari","orcid":"0000-0001-8254-4957","position":15,"is_corresponding":false},{"id":60883,"name":"Herbert M Sauro","orcid":"0000-0002-3659-6817","position":16,"is_corresponding":false},{"id":60821,"name":"Lucian P. Smith","orcid":"0000-0001-7002-6386","position":0,"is_corresponding":true}],"reference_count":29,"raw_metadata":null,"created_at":"2026-07-19T02:59:11.978098Z","pmid":"41343683","pmcid":"PMC12677764","fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":null,"license":null,"views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":72.2222,"fair_a":81.25,"fair_i":80.0,"fair_r":37.5,"fair_zscore":1.3577,"fair_rationale":{"fair_score":68.75,"has_llm":true,"taxonomy_version":"fair_taxonomy_v5","dimensions":{"F":{"name":"Findable","score":72.22,"criteria":[{"key":"f_dataset_pid","label":"Persistent identifier for the data","kind":"llm","weight":2.0,"fraction":0.5,"verdict":"partial","evidence":"https://biomodels.org/","grounded":true,"rationale":"The only identifier given is a URL, not a persistent identifier scheme (DOI, Handle, ARK, etc.).","anchors":["RDA-F1-01D — FAIR Data Maturity Model: 'Data is identified by a persistent identifier' (priorit","RDA-F1-02D — FAIR Data Maturity Model: 'Data is identified by a globally unique identifier'","FsF-F1-02D — F-UJI/FAIRsFAIR: 'Data is assigned a persistent identifier'"],"scored":true,"signal":null},{"key":"f_repository_named","label":"Named repository","kind":"llm","weight":2.0,"fraction":1.0,"verdict":"yes","evidence":"publicly available at https://biomodels.org/","grounded":true,"rationale":"BioModels is a named repository listed in re3data/FAIRsharing. [majority verdict 'yes' (4/5 passes agreed)]","anchors":["RDA-F4-01M — FAIR Data Maturity Model: metadata is offered so it can be harvested and indexed (","NIH DMS Policy Element 4 (NOT-OD-21-014) — name the repository where data will be archived","NSTC Desirable Characteristics of Data Repositories (2022) — 'Long-Term Sustainability', 'Reten"],"scored":true,"signal":null},{"key":"f_data_availability_statement","label":"Data-availability statement","kind":"llm","weight":2.0,"fraction":1.0,"verdict":"yes","evidence":"The updated models and files referenced in this work are publicly available at https://biomodels.org/ and code supporting this study is available at https://github.com/sys-bio/temp-biomodels and https://github.com/biosimulations/biosimulations-runutils .","grounded":true,"rationale":"The statement points to a repository (BioModels) and code repositories, which is a link to archived data in a public repository (Colavizza category 3). [majority verdict 'yes' (3/5 passes agreed)]","anchors":["Colavizza, Hrynaszkiewicz, Staden, Whitaker & McGillivray (2020), 'The citation advantage of li","Springer Nature research data policy — Data Availability Statements: standard statement templat","RDA-F3-01M — metadata clearly and explicitly includes the identifier of the data it describes"],"scored":false,"signal":null},{"key":"f_discovery_metadata","label":"Description of the dataset as an object","kind":"llm","weight":2.0,"fraction":0.5,"verdict":"partial","evidence":"All 1073 curated entries retrieved from the BioModels database have been updated, with every file now checked for correctness or replaced.","grounded":true,"rationale":"The dataset description is given in running prose rather than an itemised inventory (section, table, or list). [majority verdict 'partial' (4/5 passes agreed)]","anchors":["RDA-F2-01M — 'Rich metadata is provided to allow discovery' (priority Essential)","FsF-F2-01M — F-UJI: 'Metadata includes descriptive core elements to support data findability'","FsF-R1-01MD — F-UJI: 'Metadata specifies the content of the data'"],"scored":false,"signal":null},{"key":"f_dataset_cited","label":"Dataset formally cited","kind":"llm","weight":1.0,"fraction":0.5,"verdict":"partial","evidence":"Data Availability The updated models and files referenced in this work are publicly available at https://biomodels.org/","grounded":true,"rationale":"The dataset link appears only in the body text (data availability statement), not in the reference list.","anchors":["FORCE11 Joint Declaration of Data Citation Principles (2014) — data should be cited as a first-","RDA-F3-01M — metadata clearly and explicitly includes the identifier of the data it describes","FsF-F3-01M — F-UJI: 'Metadata includes the identifier of the data it describes'"],"scored":true,"signal":null}]},"A":{"name":"Accessible","score":81.25,"criteria":[{"key":"a_data_openly_accessible","label":"Access route free of preconditions","kind":"llm","weight":2.0,"fraction":1.0,"verdict":"yes","evidence":"The updated models and files referenced in this work are publicly available at https://biomodels.org/","grounded":true,"rationale":"The data are stated to be publicly available with no precondition.","anchors":["RDA-A1.1-01D — 'Data is accessible through a free access protocol'","FsF-A1-01M — F-UJI: 'Metadata contains access level and access conditions of the data'","NSTC Desirable Characteristics of Data Repositories (2022) — 'Free and Easy Access'"],"scored":true,"signal":null},{"key":"a_access_conditions_stated","label":"Access level labelled","kind":"llm","weight":1.0,"fraction":1.0,"verdict":"yes","evidence":"publicly available","grounded":true,"rationale":"The paper labels the data as 'publicly available', which is an explicit access-level label.","anchors":["FsF-A1-01M — F-UJI: 'Metadata contains access level and access conditions of the data'","RDA-A1-01M — metadata contains information to enable the user to get access to the data","COAR Controlled Vocabularies — Access Rights v1.0 (open / embargoed / restricted / metadata-onl"],"scored":false,"signal":null},{"key":"a_controlled_access_for_sensitive","label":"Gatekeeper for sensitive data","kind":"llm","weight":0.5,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"The data are computational models and not sensitive human-subject data; no gatekeeper is mentioned.","anchors":["NIH Genomic Data Sharing Policy (NOT-OD-14-124) — controlled-access via a Data Access Committee","RDA-A1.2-01D — 'Data is accessible through an access protocol that supports authentication and ","NIH DMS Policy Element 5 (NOT-OD-21-014) — Access, Distribution, or Reuse Considerations (conse"],"scored":false,"signal":null},{"key":"a_timeline_retention","label":"Availability timing & retention","kind":"llm","weight":0.5,"fraction":0.5,"verdict":"partial","evidence":"The updated models and files referenced in this work are publicly available at https://biomodels.org/","grounded":true,"rationale":"The paper states the data are available now but does not mention how long they will persist. [majority verdict 'partial' (3/5 passes agreed)]","anchors":["NIH DMS Plan Element 4 (NOT-OD-21-014) — Data Preservation, Access, and Associated Timelines","NSTC Desirable Characteristics (2022), Organizational Infrastructure: 'Retention Policy'","RDA-A2-01M — 'Metadata is guaranteed to remain available after data is no longer available'"],"scored":false,"signal":null}]},"I":{"name":"Interoperable","score":80.0,"criteria":[{"key":"i_open_nonproprietary_format","label":"Open file format","kind":"llm","weight":1.0,"fraction":1.0,"verdict":"yes","evidence":"Systems Biology Markup Language (SBML)","grounded":true,"rationale":"SBML, SED-ML, and HDF5 are open, non-proprietary community-standard formats.","anchors":["FsF-R1.3-02D — F-UJI: 'Data is available in a file format recommended by the target research co","RDA-R1.3-02D — data is expressed in a machine-understandable community standard","RDA-I1-01D — data uses a knowledge representation expressed in a standardised format"],"scored":true,"signal":null},{"key":"i_community_standard_vocabulary","label":"Community standard / vocabulary","kind":"llm","weight":1.0,"fraction":1.0,"verdict":"yes","evidence":"Systems Biology Markup Language (SBML)","grounded":true,"rationale":"The paper names SBML, SED-ML, MIRIAM, and KiSAO, all of which are community data/metadata standards. [majority verdict 'yes' (4/5 passes agreed)]","anchors":["RDA-R1.3-01M — 'Metadata complies with a community standard' (priority Essential)","RDA-R1.3-01D — 'Data complies with a community standard'","RDA-I2-01M — '(Meta)data use vocabularies that follow FAIR principles'"],"scored":false,"signal":null},{"key":"i_qualified_references","label":"Identifiers for the resources the data depend on","kind":"llm","weight":0.5,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"The paper does not provide an identifier (DOI, accession, etc.) for any external resource its data depends on; it only gives bare names or general URLs. [majority verdict 'no' (3/5 passes agreed)]","anchors":["RDA-I3-01M — '(meta)data include references to other (meta)data'","RDA-I3-03M — 'metadata includes qualified references to other metadata'","FsF-I3-01M — F-UJI: 'Metadata includes links between the data and its related entities'"],"scored":false,"signal":null}]},"R":{"name":"Reusable","score":37.5,"criteria":[{"key":"r_reuse_license","label":"Reuse licence","kind":"llm","weight":2.0,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"The paper does not state a reuse license for the data; the CC BY notice applies to the article only.","anchors":["RDA-R1.1-01M — 'Metadata includes information about the licence under which the data can be reu","RDA-R1.1-02M — 'Metadata refers to a standard reuse licence'","RDA-R1.1-03M — 'Metadata refers to a machine-understandable reuse licence'"],"scored":true,"signal":null},{"key":"r_provenance_methods","label":"Provenance of the data","kind":"llm","weight":1.0,"fraction":0.5,"verdict":"partial","evidence":"COPASI [25], Tellurium [26], VCell [27,28], PySCeS [29] and Amici [30]","grounded":false,"rationale":"The paper names specific simulation engines (COPASI, Tellurium, etc.) used to produce the data. [downgraded to 'partial' — no verifiable quote from the paper] [majority verdict 'partial' (3/5 passes agreed)]","anchors":["RDA-R1.2-01M — 'Metadata includes provenance information according to community- specific standa","FsF-R1.2-01M — F-UJI: 'Metadata includes provenance information about data creation or generati","W3C PROV-O (W3C Recommendation, 2013) — the entity/activity/agent model of provenance"],"scored":false,"signal":null},{"key":"r_documentation_codebook","label":"Documentation / codebook","kind":"llm","weight":1.0,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No separate documentation object (README, codebook, data dictionary) is mentioned as accompanying the data.","anchors":["RDA-R1-01M — '(Meta)data are richly described with a plurality of accurate and relevant attribu","FsF-R1-01MD — F-UJI: 'Metadata specifies the content of the data'","NIH DMS Policy Element 3 (NOT-OD-21-014) — Standards (documentation and metadata to accompany t"],"scored":false,"signal":null},{"key":"r_versioning","label":"Snapshot identified","kind":"llm","weight":0.5,"fraction":0.5,"verdict":"partial","evidence":"we took a snapshot in time from June of 2024 of the curated branch of BioModels","grounded":true,"rationale":"The paper provides a date (June 2024) as the snapshot reference, but no version token.","anchors":["DataCite Metadata Schema 4.6 — the 'Version' property","RDA-R1.2-01M — provenance information (which version was used is provenance)","NSTC Desirable Characteristics of Data Repositories (2022) — 'Provenance', 'Retention Policy'"],"scored":true,"signal":null},{"key":"x_code_availability","label":"Analysis code available","kind":"llm","weight":1.0,"fraction":1.0,"verdict":"yes","evidence":"code supporting this study is available at https://github.com/sys-bio/temp-biomodels and https://github.com/biosimulations/biosimulations-runutils","grounded":true,"rationale":"The paper gives machine-resolvable GitHub URLs for the code.","anchors":["NIH DMS Policy Element 2 (NOT-OD-21-014) — 'Related Tools, Software and/or Code'","FAIR4RS Principles v1.0 (Chue Hong et al., 2022; RDA/FORCE11/ReSA) — FAIR Principles for Resear","FORCE11 Software Citation Principles (Smith, Katz & Niemeyer, 2016, PeerJ CS 2:e86)"],"scored":true,"signal":null},{"key":"x_funding_attribution","label":"Funder and award number","kind":"llm","weight":0.5,"fraction":1.0,"verdict":"yes","evidence":"NIH grant P41EB023912","grounded":true,"rationale":"The paper provides specific grant numbers (P41EB023912, R24GM137787, 116030) attached to named funders.","anchors":["DataCite Metadata Schema 4.6 — 'FundingReference' property (funderName, funderIdentifier, award","Crossref Funder Registry — canonical funder identifiers for funding metadata","RDA-F2-01M — rich metadata provided to allow discovery (funding is part of the descriptive reco"],"scored":true,"signal":null}]}},"actions":[{"key":"r_reuse_license","dimension":"R","label":"Reuse licence","action":"Attach a standard, machine-readable open licence to the deposit — CC0 or CC BY, which is what Horizon Europe and most funders expect — and print the licence identifier in the paper. 'Free to use' is not a licence: it grants nothing a reuser's institution can rely on.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"The paper does not state a reuse license for the data; the CC BY notice applies to the article only.","gain":16.67,"priority":"essential","scored":true},{"key":"f_dataset_pid","dimension":"F","label":"Persistent identifier for the data","action":"Mint or cite a persistent identifier for the dataset — a repository DOI or an accession from a registered repository — and print it in the paper. A bare URL is not persistent: it is the single most common cause of a dead data link five years after publication. For social science data, deposit in ICPSR or the Harvard Dataverse.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"https://biomodels.org/","why":"The only identifier given is a URL, not a persistent identifier scheme (DOI, Handle, ARK, etc.).","gain":8.33,"priority":"essential","scored":true},{"key":"f_dataset_cited","dimension":"F","label":"Dataset formally cited","action":"Cite the dataset in the reference list like a publication — creator, year, title, repository, DOI/accession — and cite it in-text where it is used. Only a reference- list entry is machine-readable to Crossref/DataCite, and only a citation lets the data earn credit. Cite the social science repository accession (e.g. from ICPSR or the Harvard Dataverse) in the reference list.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"Data Availability The updated models and files referenced in this work are publicly available at https://biomodels.org/","why":"The dataset link appears only in the body text (data availability statement), not in the reference list.","gain":4.17,"priority":"important","scored":true},{"key":"r_versioning","dimension":"R","label":"Snapshot identified","action":"Version the deposit and cite the exact version analysed (a version-specific DOI, or an accession with its version suffix). A reader reproducing your work against 'the current release' is reproducing it against a different dataset.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"we took a snapshot in time from June of 2024 of the curated branch of BioModels","why":"The paper provides a date (June 2024) as the snapshot reference, but no version token.","gain":2.08,"priority":"useful","scored":true},{"key":"f_discovery_metadata","dimension":"F","label":"Description of the dataset as an object","action":"Add a 'Data Records' section: itemise every file in the deposit and every variable or sample it holds, with counts and units. Describe the dataset as an object in its own right, not as a by-product of the findings — this is what makes it discoverable to someone who is not looking for your paper.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"All 1073 curated entries retrieved from the BioModels database have been updated, with every file now checked for correctness or replaced.","why":"The dataset description is given in running prose rather than an itemised inventory (section, table, or list). [majority verdict 'partial' (4/5 passes agreed)]","gain":0.0,"priority":"essential","scored":false},{"key":"r_provenance_methods","dimension":"R","label":"Provenance of the data","action":"Name the instruments, kits, and software — with versions — that produced the data, not just the verbs. 'Reads were aligned' is not provenance; 'aligned with STAR v2.7.9a to GRCh38' is, because someone else can rerun it.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"COPASI [25], Tellurium [26], VCell [27,28], PySCeS [29] and Amici [30]","why":"The paper names specific simulation engines (COPASI, Tellurium, etc.) used to produce the data. [downgraded to 'partial' — no verifiable quote from the paper] [majority verdict 'partial' (3/5 passes agreed)]","gain":0.0,"priority":"important","scored":false},{"key":"r_documentation_codebook","dimension":"R","label":"Documentation / codebook","action":"Ship a README and a data dictionary IN the deposit — every file, every variable, its units, its allowed values, its missing-value codes. It is the cheapest single thing that makes a dataset usable by someone who was not in the lab, and a table buried in the article does not travel with the data.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No separate documentation object (README, codebook, data dictionary) is mentioned as accompanying the data.","gain":0.0,"priority":"important","scored":false},{"key":"a_controlled_access_for_sensitive","dimension":"A","label":"Gatekeeper for sensitive data","action":"Route sensitive data through an institutional gatekeeper — deposit in a controlled- access repository (dbGaP, EGA) with a Data Access Committee and a published DUA — rather than through the corresponding author's inbox. An author-gated dataset dies with the author's email address, and 'on reasonable request' has been shown repeatedly not to yield data.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"The data are computational models and not sensitive human-subject data; no gatekeeper is mentioned.","gain":0.0,"priority":"useful","scored":false},{"key":"i_qualified_references","dimension":"I","label":"Identifiers for the resources the data depend on","action":"Cite by identifier every resource the data depend on — the source datasets' accessions, the reference build (GRCh38 / GCA_000001405.28), the cohort application number, the code DOI — and register those relations on the dataset record (IsDerivedFrom, IsSupplementTo). A name is not a link: it cannot be resolved, versioned, or followed by a machine.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"The paper does not provide an identifier (DOI, accession, etc.) for any external resource its data depends on; it only gives bare names or general URLs. [majority verdict 'no' (3/5 passes agreed)]","gain":0.0,"priority":"useful","scored":false},{"key":"a_timeline_retention","dimension":"A","label":"Availability timing & retention","action":"State when the data become available AND how long they will be retained — cite the repository's preservation policy. NIH DMS Element 4 asks for both; most papers give neither.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"The updated models and files referenced in this work are publicly available at https://biomodels.org/","why":"The paper states the data are available now but does not mention how long they will persist. [majority verdict 'partial' (3/5 passes agreed)]","gain":0.0,"priority":"useful","scored":false}],"suggestions":["Attach a standard, machine-readable open licence to the deposit — CC0 or CC BY, which is what Horizon Europe and most funders expect — and print the licence identifier in the paper. 'Free to use' is not a licence: it grants nothing a reuser's institution can rely on.","Mint or cite a persistent identifier for the dataset — a repository DOI or an accession from a registered repository — and print it in the paper. A bare URL is not persistent: it is the single most common cause of a dead data link five years after publication. For social science data, deposit in ICPSR or the Harvard Dataverse.","Cite the dataset in the reference list like a publication — creator, year, title, repository, DOI/accession — and cite it in-text where it is used. Only a reference- list entry is machine-readable to Crossref/DataCite, and only a citation lets the data earn credit. Cite the social science repository accession (e.g. from ICPSR or the Harvard Dataverse) in the reference list.","Version the deposit and cite the exact version analysed (a version-specific DOI, or an accession with its version suffix). A reader reproducing your work against 'the current release' is reproducing it against a different dataset.","Add a 'Data Records' section: itemise every file in the deposit and every variable or sample it holds, with counts and units. Describe the dataset as an object in its own right, not as a by-product of the findings — this is what makes it discoverable to someone who is not looking for your paper."],"model":"deepseek/deepseek-v4-flash","agent_version":"fair_agent_v8","fulltext_source":"epmc_xml"},"fair_model":"deepseek/deepseek-v4-flash","fair_agent_version":"fair_agent_v8","fair_fulltext_source":"epmc_xml","fair_has_llm":true,"fair_computed_at":"2026-07-20T13:51:29.810557Z","clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}