{"doi":"10.1016/j.xops.2025.101050","title":"BEnchmarking Large Language Models for Ophthalmology (BELO): An Expert-Curated Data Set and Evaluation Framework for Knowledge and Reasoning","abstract":"Purpose: Current benchmarks evaluating large language models (LLMs) in ophthalmology are narrow and disproportionately prioritize accuracy. We introduce BEnchmarking LLMs for Ophthalmology (BELO), a standardized evaluation benchmark developed through multiple rounds of expert checking by 13 ophthalmologists. BEnchmarking LLMs for Ophthalmology assesses ophthalmology-related knowledge and reasoning quality. Subjects: This study did not involve human participation. Design: Cross-sectional study. Methods: Using keyword matching and a fine-tuned PubMed Bidirectional Encoder Representations from Transformers model, we curated ophthalmology-specific multiple-choice questions (MCQs) from diverse medical data sets (Basic and Clinical Science Course [BCSC], Multi-Subject Multi-Choice Dataset for Medical domain [MedMCQA], Medical Question Answering [MedQA], Biomedical Semantic Indexing and Question Answering [BioASQ], and PubMed Question Answering [PubMedQA]). The data set underwent multiple rounds of expert checking. Duplicate and substandard questions were systematically removed. Ten ophthalmologists refined the explanations of each MCQ's correct answer. This was further adjudicated by 3 senior ophthalmologists. To illustrate BELO's utility, we evaluated 8 LLMs (OpenAI o1, o3-mini, GPT-5, GPT-4o, DeepSeek-R1, MedGemma-4B, Llama-3-8B, and Gemini 1.5 Pro). Main Outcome Measures: The 8 LLMs were evaluated in terms using accuracy, macro-F1, and 5 text-generation metrics (Recall-Oriented Understudy for Gisting Evaluation, BERTScore, BARTScore, Metric for Evaluation of Translation with Explicit Ordering, and AlignScore). In a further evaluation involving human experts, 2 ophthalmologists qualitatively reviewed 50 randomly selected outputs for accuracy, comprehensiveness, and completeness. Results: BEnchmarking LLMs for Ophthalmology consists of 900 high-quality, expert-reviewed questions aggregated from 5 sources: BCSC (260), BioASQ (10), MedMCQA (572), MedQA (40), and PubMedQA (18). To demonstrate BELO's utility, we conducted a series of benchmarking exercises. In the quantitative evaluation, GPT-5 achieved the highest accuracy (0.90, 95% confidence interval [CI]: 0.89-0.92) and macro-F1 score (0.91, 95% CI: 0.89-0.93). On the other hand, the models' performance on text-generation metrics varied and were generally suboptimal, with scores ranging from 20.4 to 72.0 (out of 100, excluding the BARTScore metric), indicating room for improvement in clinical reasoning. In expert evaluations, GPT-4o was rated highest for accuracy and readability, while Gemini 1.5 Pro scored highest for completeness. A public leaderboard has been established to promote transparent evaluation and reporting. Importantly, the BELO data set will remain a hold-out, evaluation-only benchmark to ensure fair and reproducible comparisons of future models. Conclusions: BEnchmarking LLMs for Ophthalmology provides a robust clinically relevant benchmark for evaluating both the accuracy and reasoning capabilities of current and emerging LLMs in ophthalmology. Future BELO benchmarking efforts will expand to include vision-based question answering and clinical scenario management tasks. Financial Disclosures: The authors have no proprietary or commercial interest in any materials discussed in this article.","journal":"Ophthalmology Science","year":2025,"id":549758,"datarank":0.10397207708399181,"base_score":0.6931471805599453,"endowment":0.6931471805599453,"self_citation_contribution":0.10397207708399181,"citation_network_contribution":0.0,"self_endowment_contribution":0.10397207708399181,"citer_contribution":0.0,"corpus_percentile":22.178386323199504,"corpus_rank":9377,"citation_count":1,"citer_count":1,"citers_with_citation_signal":0,"citers_with_endowment":0,"datacite_reuse_total":0,"is_dataset":true,"is_dataset_confidence":0.9123,"is_data_producer":false,"deposit_databanks":null,"is_oa":true,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":"2025-01-01","fair_score":20.8333,"fair_percentile":36.38031183124427,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":1152765,"name":"Xuguang Ai","orcid":"0009-0000-2432-8316","position":1,"is_corresponding":false},{"id":1444977,"name":"Thaddaeus Wai Soon Lo","orcid":null,"position":2,"is_corresponding":false},{"id":983000,"name":"Aidan Gilson","orcid":"0000-0002-4770-4705","position":3,"is_corresponding":false},{"id":1444978,"name":"Minjie Zou","orcid":null,"position":4,"is_corresponding":false},{"id":1444491,"name":"Ke Zou","orcid":"0000-0002-2025-9859","position":5,"is_corresponding":false},{"id":1345241,"name":"Hyunjae Kim","orcid":"0000-0001-9749-266X","position":6,"is_corresponding":false},{"id":794328,"name":"Mingjia Yang","orcid":"0000-0003-0739-3989","position":7,"is_corresponding":false},{"id":1444492,"name":"Krithi Pushpanathan","orcid":"0000-0001-6263-8489","position":8,"is_corresponding":false},{"id":1444979,"name":"Samantha Min Er Yew","orcid":null,"position":9,"is_corresponding":false},{"id":1444980,"name":"Wan Ting Loke","orcid":null,"position":10,"is_corresponding":false},{"id":854779,"name":"Jocelyn Hui Lin Goh","orcid":"0000-0002-5052-6081","position":11,"is_corresponding":false},{"id":1444981,"name":"Yibing Chen","orcid":null,"position":12,"is_corresponding":false},{"id":1444493,"name":"Yiming Kong","orcid":"0000-0002-5223-7152","position":13,"is_corresponding":false},{"id":1444982,"name":"Evelyn X. Fu","orcid":null,"position":14,"is_corresponding":false},{"id":1444983,"name":"Michelle Ong","orcid":null,"position":15,"is_corresponding":false},{"id":340506,"name":"Kristen Nwanyanwu","orcid":"0000-0003-1146-7733","position":16,"is_corresponding":false},{"id":854787,"name":"Amisha Dave","orcid":"0000-0001-8377-8309","position":17,"is_corresponding":false},{"id":1431280,"name":"Kelvin Zhenghao Li","orcid":"0000-0002-9993-5118","position":18,"is_corresponding":false},{"id":1444984,"name":"Chen‐Hsin Sun","orcid":null,"position":19,"is_corresponding":false},{"id":869758,"name":"Mark A. Chia","orcid":"0000-0003-0339-5186","position":20,"is_corresponding":false},{"id":1444985,"name":"Gabriel Dawei Yang","orcid":null,"position":21,"is_corresponding":false},{"id":1367098,"name":"Wendy Wong","orcid":"0000-0002-4514-250X","position":22,"is_corresponding":false},{"id":76007,"name":"David Ziyou Chen","orcid":"0000-0002-2153-3100","position":23,"is_corresponding":false},{"id":11486,"name":"Dianbo Liu","orcid":"0000-0002-3042-9161","position":24,"is_corresponding":false},{"id":745733,"name":"Maxwell Singer","orcid":"0000-0001-9583-3846","position":25,"is_corresponding":false},{"id":1209631,"name":"Fares Antaki","orcid":"0000-0001-6679-7276","position":26,"is_corresponding":false},{"id":335371,"name":"Lucian V. Del Priore","orcid":null,"position":27,"is_corresponding":false},{"id":22574,"name":"Jost B. Jonas","orcid":"0000-0003-2972-5227","position":28,"is_corresponding":false},{"id":1156055,"name":"Ron A. Adelman","orcid":"0000-0003-4049-3551","position":29,"is_corresponding":false},{"id":236881,"name":"Qingyu Chen","orcid":"0000-0002-6036-1516","position":30,"is_corresponding":false},{"id":1444986,"name":"Yih-Chung Tham","orcid":null,"position":31,"is_corresponding":false},{"id":1444976,"name":"Sahana Srinivasan","orcid":null,"position":0,"is_corresponding":true}],"reference_count":28,"raw_metadata":null,"created_at":"2026-07-19T02:54:12.321988Z","pmid":"41696659","pmcid":"PMC12906013","fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":null,"license":null,"views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":33.3333,"fair_a":56.25,"fair_i":60.0,"fair_r":25.0,"fair_zscore":-0.5389,"fair_rationale":{"fair_score":20.83,"has_llm":true,"taxonomy_version":"fair_taxonomy_v5","dimensions":{"F":{"name":"Findable","score":33.33,"criteria":[{"key":"f_dataset_pid","label":"Persistent identifier for the data","kind":"llm","weight":2.0,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No persistent identifier (DOI, Handle, ARK, URN, or repository accession) is given for the dataset; the only URL is for the leaderboard, not the data. [majority verdict 'no' (3/5 passes agreed)]","anchors":["RDA-F1-01D — FAIR Data Maturity Model: 'Data is identified by a persistent identifier' (priorit","RDA-F1-02D — FAIR Data Maturity Model: 'Data is identified by a globally unique identifier'","FsF-F1-02D — F-UJI/FAIRsFAIR: 'Data is assigned a persistent identifier'"],"scored":true,"signal":null},{"key":"f_repository_named","label":"Named repository","kind":"llm","weight":2.0,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No repository is named as the holder; the data are held by the authors and accessed via a data usage agreement.","anchors":["RDA-F4-01M — FAIR Data Maturity Model: metadata is offered so it can be harvested and indexed (","NIH DMS Policy Element 4 (NOT-OD-21-014) — name the repository where data will be archived","NSTC Desirable Characteristics of Data Repositories (2022) — 'Long-Term Sustainability', 'Reten"],"scored":true,"signal":null},{"key":"f_data_availability_statement","label":"Data-availability statement","kind":"llm","weight":2.0,"fraction":0.5,"verdict":"partial","evidence":"Data Availability All data can be provided upon reasonable request and evaluation.","grounded":true,"rationale":"The statement points to the authors (request) rather than to a repository record, placing it in Colavizza category 1.","anchors":["Colavizza, Hrynaszkiewicz, Staden, Whitaker & McGillivray (2020), 'The citation advantage of li","Springer Nature research data policy — Data Availability Statements: standard statement templat","RDA-F3-01M — metadata clearly and explicitly includes the identifier of the data it describes"],"scored":false,"signal":null},{"key":"f_discovery_metadata","label":"Description of the dataset as an object","kind":"llm","weight":2.0,"fraction":1.0,"verdict":"yes","evidence":"Table 1 Compositions of the Curated BELO Data Set","grounded":true,"rationale":"The paper provides an itemized inventory of the dataset's composition in a table, which is a structural text object. [majority verdict 'yes' (4/5 passes agreed)]","anchors":["RDA-F2-01M — 'Rich metadata is provided to allow discovery' (priority Essential)","FsF-F2-01M — F-UJI: 'Metadata includes descriptive core elements to support data findability'","FsF-R1-01MD — F-UJI: 'Metadata specifies the content of the data'"],"scored":false,"signal":null},{"key":"f_dataset_cited","label":"Dataset formally cited","kind":"llm","weight":1.0,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No identifier for the dataset appears anywhere in the text, neither in the reference list nor in the body. [majority verdict 'no' (3/5 passes agreed)]","anchors":["FORCE11 Joint Declaration of Data Citation Principles (2014) — data should be cited as a first-","RDA-F3-01M — metadata clearly and explicitly includes the identifier of the data it describes","FsF-F3-01M — F-UJI: 'Metadata includes the identifier of the data it describes'"],"scored":true,"signal":null}]},"A":{"name":"Accessible","score":56.25,"criteria":[{"key":"a_data_openly_accessible","label":"Access route free of preconditions","kind":"llm","weight":2.0,"fraction":0.5,"verdict":"partial","evidence":"Contingent on signing a data usage agreement, stipulating that they would use BELO for validation only, we will then pass the questions to them but without the ground truth answers.","grounded":true,"rationale":"The text specifies a followable access process with a precondition (signing a data usage agreement), which is a defined, institutional step. [majority verdict 'partial' (3/5 passes agreed)]","anchors":["RDA-A1.1-01D — 'Data is accessible through a free access protocol'","FsF-A1-01M — F-UJI: 'Metadata contains access level and access conditions of the data'","NSTC Desirable Characteristics of Data Repositories (2022) — 'Free and Easy Access'"],"scored":true,"signal":null},{"key":"a_access_conditions_stated","label":"Access level labelled","kind":"llm","weight":1.0,"fraction":0.5,"verdict":"partial","evidence":"All data can be provided upon reasonable request and evaluation.","grounded":true,"rationale":"The text describes an access action (request and evaluation) but does not apply an explicit access-level label from the standard vocabulary.","anchors":["FsF-A1-01M — F-UJI: 'Metadata contains access level and access conditions of the data'","RDA-A1-01M — metadata contains information to enable the user to get access to the data","COAR Controlled Vocabularies — Access Rights v1.0 (open / embargoed / restricted / metadata-onl"],"scored":false,"signal":null},{"key":"a_controlled_access_for_sensitive","label":"Gatekeeper for sensitive data","kind":"llm","weight":0.5,"fraction":0.5,"verdict":"partial","evidence":"All data can be provided upon reasonable request and evaluation.","grounded":true,"rationale":"The gatekeeper is a natural person (the authors), as the data are obtainable only by asking them. [majority verdict 'partial' (3/5 passes agreed)]","anchors":["NIH Genomic Data Sharing Policy (NOT-OD-14-124) — controlled-access via a Data Access Committee","RDA-A1.2-01D — 'Data is accessible through an access protocol that supports authentication and ","NIH DMS Policy Element 5 (NOT-OD-21-014) — Access, Distribution, or Reuse Considerations (conse"],"scored":false,"signal":null},{"key":"a_timeline_retention","label":"Availability timing & retention","kind":"llm","weight":0.5,"fraction":1.0,"verdict":"yes","evidence":"the BELO data set will remain a hold-out, evaluation-only benchmark to ensure fair and reproducible comparisons of future models.","grounded":true,"rationale":"The text states that the dataset will remain available indefinitely as a hold-out benchmark, implying a persistence commitment.","anchors":["NIH DMS Plan Element 4 (NOT-OD-21-014) — Data Preservation, Access, and Associated Timelines","NSTC Desirable Characteristics (2022), Organizational Infrastructure: 'Retention Policy'","RDA-A2-01M — 'Metadata is guaranteed to remain available after data is no longer available'"],"scored":false,"signal":null}]},"I":{"name":"Interoperable","score":60.0,"criteria":[{"key":"i_open_nonproprietary_format","label":"Open file format","kind":"llm","weight":1.0,"fraction":1.0,"verdict":"yes","evidence":"These data sets were then standardized and formatted into JavaScript Object Notation files","grounded":true,"rationale":"JSON is an open, community-standard format.","anchors":["FsF-R1.3-02D — F-UJI: 'Data is available in a file format recommended by the target research co","RDA-R1.3-02D — data is expressed in a machine-understandable community standard","RDA-I1-01D — data uses a knowledge representation expressed in a standardised format"],"scored":true,"signal":null},{"key":"i_community_standard_vocabulary","label":"Community standard / vocabulary","kind":"llm","weight":1.0,"fraction":0.5,"verdict":"partial","evidence":"Our study adheres to the Declaration of Helsinki and follows the Strengthening the Reporting of Observational Studies in Epidemiology reporting guideline.","grounded":true,"rationale":"The only named standard is a manuscript reporting guideline (STROBE), not a data or metadata community standard. [majority verdict 'partial' (4/5 passes agreed)]","anchors":["RDA-R1.3-01M — 'Metadata complies with a community standard' (priority Essential)","RDA-R1.3-01D — 'Data complies with a community standard'","RDA-I2-01M — '(Meta)data use vocabularies that follow FAIR principles'"],"scored":false,"signal":null},{"key":"i_qualified_references","label":"Identifiers for the resources the data depend on","kind":"llm","weight":0.5,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No identifier (accession, DOI, RRID, etc.) is given for any external resource that the data depend on or derive from. [majority verdict 'no' (4/5 passes agreed)]","anchors":["RDA-I3-01M — '(meta)data include references to other (meta)data'","RDA-I3-03M — 'metadata includes qualified references to other metadata'","FsF-I3-01M — F-UJI: 'Metadata includes links between the data and its related entities'"],"scored":false,"signal":null}]},"R":{"name":"Reusable","score":25.0,"criteria":[{"key":"r_reuse_license","label":"Reuse licence","kind":"llm","weight":2.0,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No licence or terms document is named for the data; the CC BY-NC-ND licence applies to the article, not the data. [majority verdict 'no' (3/5 passes agreed)]","anchors":["RDA-R1.1-01M — 'Metadata includes information about the licence under which the data can be reu","RDA-R1.1-02M — 'Metadata refers to a standard reuse licence'","RDA-R1.1-03M — 'Metadata refers to a machine-understandable reuse licence'"],"scored":true,"signal":null},{"key":"r_provenance_methods","label":"Provenance of the data","kind":"llm","weight":1.0,"fraction":0.5,"verdict":"partial","evidence":"We extracted ophthalmic QA pairs from these data sets using 2 approaches: keyword matching and a fine-tuned PubMedBERT-based approach","grounded":false,"rationale":"The paper names the specific tool (PubMedBERT) used to produce the data. [downgraded to 'partial' — no verifiable quote from the paper] [majority verdict 'partial' (2/5 passes agreed)]","anchors":["RDA-R1.2-01M — 'Metadata includes provenance information according to community- specific standa","FsF-R1.2-01M — F-UJI: 'Metadata includes provenance information about data creation or generati","W3C PROV-O (W3C Recommendation, 2013) — the entity/activity/agent model of provenance"],"scored":false,"signal":null},{"key":"r_documentation_codebook","label":"Documentation / codebook","kind":"llm","weight":1.0,"fraction":0.5,"verdict":"partial","evidence":"Table 1 Compositions of the Curated BELO Data Set","grounded":true,"rationale":"Variable-level definitions are provided inside the article (Table 1), but no documentation object is said to accompany the data. [majority verdict 'partial' (4/5 passes agreed)]","anchors":["RDA-R1-01M — '(Meta)data are richly described with a plurality of accurate and relevant attribu","FsF-R1-01MD — F-UJI: 'Metadata specifies the content of the data'","NIH DMS Policy Element 3 (NOT-OD-21-014) — Standards (documentation and metadata to accompany t"],"scored":false,"signal":null},{"key":"r_versioning","label":"Snapshot identified","kind":"llm","weight":0.5,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No version token or date is given to pin the snapshot of the dataset.","anchors":["DataCite Metadata Schema 4.6 — the 'Version' property","RDA-R1.2-01M — provenance information (which version was used is provenance)","NSTC Desirable Characteristics of Data Repositories (2022) — 'Provenance', 'Retention Policy'"],"scored":true,"signal":null},{"key":"x_code_availability","label":"Analysis code available","kind":"llm","weight":1.0,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No code location is provided; the study's own code is not mentioned beyond the use of Python, and no repository or DOI is given.","anchors":["NIH DMS Policy Element 2 (NOT-OD-21-014) — 'Related Tools, Software and/or Code'","FAIR4RS Principles v1.0 (Chue Hong et al., 2022; RDA/FORCE11/ReSA) — FAIR Principles for Resear","FORCE11 Software Citation Principles (Smith, Katz & Niemeyer, 2016, PeerJ CS 2:e86)"],"scored":true,"signal":null},{"key":"x_funding_attribution","label":"Funder and award number","kind":"llm","weight":0.5,"fraction":1.0,"verdict":"yes","evidence":"4R00LM014024","grounded":true,"rationale":"An award number is given for a funder (NIH/NLM), satisfying the requirement for an alphanumeric grant identifier.","anchors":["DataCite Metadata Schema 4.6 — 'FundingReference' property (funderName, funderIdentifier, award","Crossref Funder Registry — canonical funder identifiers for funding metadata","RDA-F2-01M — rich metadata provided to allow discovery (funding is part of the descriptive reco"],"scored":true,"signal":null}]}},"actions":[{"key":"f_dataset_pid","dimension":"F","label":"Persistent identifier for the data","action":"Mint or cite a persistent identifier for the dataset — a repository DOI or an accession from a registered repository — and print it in the paper. A bare URL is not persistent: it is the single most common cause of a dead data link five years after publication. For chemistry / materials data, deposit in Zenodo, PubChem or the Cambridge Structural Database (CSD).","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No persistent identifier (DOI, Handle, ARK, URN, or repository accession) is given for the dataset; the only URL is for the leaderboard, not the data. [majority verdict 'no' (3/5 passes agreed)]","gain":16.67,"priority":"essential","scored":true},{"key":"f_repository_named","dimension":"F","label":"Named repository","action":"Deposit the data in a repository registered in re3data/FAIRsharing (a domain repository such as GEO, SRA, dbGaP, PRIDE, or a generalist such as Zenodo, Dryad, Dataverse) and name it explicitly in the paper. A lab website is not an archive: it has no retention commitment and no accession. For chemistry / materials data, deposit in Zenodo, PubChem or the Cambridge Structural Database (CSD).","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No repository is named as the holder; the data are held by the authors and accessed via a data usage agreement.","gain":16.67,"priority":"essential","scored":true},{"key":"r_reuse_license","dimension":"R","label":"Reuse licence","action":"Attach a standard, machine-readable open licence to the deposit — CC0 or CC BY, which is what Horizon Europe and most funders expect — and print the licence identifier in the paper. 'Free to use' is not a licence: it grants nothing a reuser's institution can rely on.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No licence or terms document is named for the data; the CC BY-NC-ND licence applies to the article, not the data. [majority verdict 'no' (3/5 passes agreed)]","gain":16.67,"priority":"essential","scored":true},{"key":"a_data_openly_accessible","dimension":"A","label":"Access route free of preconditions","action":"Remove the precondition or justify it. Release the data at publication with no embargo, no registration wall, and no approval step — NIH's zero-embargo public- access rule (NOT-OD-25-101) has already made 'available at publication' the federal baseline for the article; the data should not lag behind it. For chemistry / materials data, deposit in Zenodo, PubChem or the Cambridge Structural Database (CSD).","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"Contingent on signing a data usage agreement, stipulating that they would use BELO for validation only, we will then pass the questions to them but without the ground truth answers.","why":"The text specifies a followable access process with a precondition (signing a data usage agreement), which is a defined, institutional step. [majority verdict 'partial' (3/5 passes agreed)]","gain":8.33,"priority":"essential","scored":true},{"key":"f_dataset_cited","dimension":"F","label":"Dataset formally cited","action":"Cite the dataset in the reference list like a publication — creator, year, title, repository, DOI/accession — and cite it in-text where it is used. Only a reference- list entry is machine-readable to Crossref/DataCite, and only a citation lets the data earn credit. Cite the chemistry / materials repository accession (e.g. from Zenodo, PubChem or the Cambridge Structural Database (CSD)) in the reference list.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No identifier for the dataset appears anywhere in the text, neither in the reference list nor in the body. [majority verdict 'no' (3/5 passes agreed)]","gain":8.33,"priority":"important","scored":true},{"key":"x_code_availability","dimension":"R","label":"Analysis code available","action":"Publish the analysis code in a public forge, archive a tagged release with a DOI (Zenodo/Software Heritage), and cite that DOI in the paper. NIH DMS Element 2 asks for the tools and code, not only the data — and 'available on request' is not a locator. Archive the analysis code in a versioned repository (GitHub + a Zenodo release DOI).","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No code location is provided; the study's own code is not mentioned beyond the use of Python, and no repository or DOI is given.","gain":8.33,"priority":"important","scored":true},{"key":"r_versioning","dimension":"R","label":"Snapshot identified","action":"Version the deposit and cite the exact version analysed (a version-specific DOI, or an accession with its version suffix). A reader reproducing your work against 'the current release' is reproducing it against a different dataset.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No version token or date is given to pin the snapshot of the dataset.","gain":4.17,"priority":"useful","scored":true},{"key":"f_data_availability_statement","dimension":"F","label":"Data-availability statement","action":"Replace the statement with the repository template: name the repository and give the accession or DOI (Colavizza category 3). This is the only DAS class associated with a measured citation advantage; 'available on reasonable request' and 'within the article' are not.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"Data Availability All data can be provided upon reasonable request and evaluation.","why":"The statement points to the authors (request) rather than to a repository record, placing it in Colavizza category 1.","gain":0.0,"priority":"essential","scored":false},{"key":"a_access_conditions_stated","dimension":"A","label":"Access level labelled","action":"State the access level in words, using the standard vocabulary: 'These data are open access' / 'These data are controlled access'. A reader — and a harvester — should not have to infer the access level from the presence of a download link.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"All data can be provided upon reasonable request and evaluation.","why":"The text describes an access action (request and evaluation) but does not apply an explicit access-level label from the standard vocabulary.","gain":0.0,"priority":"important","scored":false},{"key":"i_community_standard_vocabulary","dimension":"I","label":"Community standard / vocabulary","action":"Adopt and NAME your domain's data standard — the minimum-information checklist, metadata schema, or ontology your community uses (MIAME/MINSEQE, ISA-Tab, BIDS, an OBO ontology, HL7 FHIR/OMOP) — and say which one you followed. A reporting checklist standardises your paper; it does nothing for your data. In chemistry / materials, describe the data with InChI, CIF or SMILES.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"Our study adheres to the Declaration of Helsinki and follows the Strengthening the Reporting of Observational Studies in Epidemiology reporting guideline.","why":"The only named standard is a manuscript reporting guideline (STROBE), not a data or metadata community standard. [majority verdict 'partial' (4/5 passes agreed)]","gain":0.0,"priority":"important","scored":false},{"key":"r_provenance_methods","dimension":"R","label":"Provenance of the data","action":"Name the instruments, kits, and software — with versions — that produced the data, not just the verbs. 'Reads were aligned' is not provenance; 'aligned with STAR v2.7.9a to GRCh38' is, because someone else can rerun it.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"We extracted ophthalmic QA pairs from these data sets using 2 approaches: keyword matching and a fine-tuned PubMedBERT-based approach","why":"The paper names the specific tool (PubMedBERT) used to produce the data. [downgraded to 'partial' — no verifiable quote from the paper] [majority verdict 'partial' (2/5 passes agreed)]","gain":0.0,"priority":"important","scored":false},{"key":"r_documentation_codebook","dimension":"R","label":"Documentation / codebook","action":"Ship a README and a data dictionary IN the deposit — every file, every variable, its units, its allowed values, its missing-value codes. It is the cheapest single thing that makes a dataset usable by someone who was not in the lab, and a table buried in the article does not travel with the data.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"Table 1 Compositions of the Curated BELO Data Set","why":"Variable-level definitions are provided inside the article (Table 1), but no documentation object is said to accompany the data. [majority verdict 'partial' (4/5 passes agreed)]","gain":0.0,"priority":"important","scored":false},{"key":"a_controlled_access_for_sensitive","dimension":"A","label":"Gatekeeper for sensitive data","action":"Route sensitive data through an institutional gatekeeper — deposit in a controlled- access repository (dbGaP, EGA) with a Data Access Committee and a published DUA — rather than through the corresponding author's inbox. An author-gated dataset dies with the author's email address, and 'on reasonable request' has been shown repeatedly not to yield data.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"All data can be provided upon reasonable request and evaluation.","why":"The gatekeeper is a natural person (the authors), as the data are obtainable only by asking them. [majority verdict 'partial' (3/5 passes agreed)]","gain":0.0,"priority":"useful","scored":false},{"key":"i_qualified_references","dimension":"I","label":"Identifiers for the resources the data depend on","action":"Cite by identifier every resource the data depend on — the source datasets' accessions, the reference build (GRCh38 / GCA_000001405.28), the cohort application number, the code DOI — and register those relations on the dataset record (IsDerivedFrom, IsSupplementTo). A name is not a link: it cannot be resolved, versioned, or followed by a machine.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No identifier (accession, DOI, RRID, etc.) is given for any external resource that the data depend on or derive from. [majority verdict 'no' (4/5 passes agreed)]","gain":0.0,"priority":"useful","scored":false}],"suggestions":["Mint or cite a persistent identifier for the dataset — a repository DOI or an accession from a registered repository — and print it in the paper. A bare URL is not persistent: it is the single most common cause of a dead data link five years after publication. For chemistry / materials data, deposit in Zenodo, PubChem or the Cambridge Structural Database (CSD).","Deposit the data in a repository registered in re3data/FAIRsharing (a domain repository such as GEO, SRA, dbGaP, PRIDE, or a generalist such as Zenodo, Dryad, Dataverse) and name it explicitly in the paper. A lab website is not an archive: it has no retention commitment and no accession. For chemistry / materials data, deposit in Zenodo, PubChem or the Cambridge Structural Database (CSD).","Attach a standard, machine-readable open licence to the deposit — CC0 or CC BY, which is what Horizon Europe and most funders expect — and print the licence identifier in the paper. 'Free to use' is not a licence: it grants nothing a reuser's institution can rely on.","Remove the precondition or justify it. Release the data at publication with no embargo, no registration wall, and no approval step — NIH's zero-embargo public- access rule (NOT-OD-25-101) has already made 'available at publication' the federal baseline for the article; the data should not lag behind it. For chemistry / materials data, deposit in Zenodo, PubChem or the Cambridge Structural Database (CSD).","Cite the dataset in the reference list like a publication — creator, year, title, repository, DOI/accession — and cite it in-text where it is used. Only a reference- list entry is machine-readable to Crossref/DataCite, and only a citation lets the data earn credit. Cite the chemistry / materials repository accession (e.g. from Zenodo, PubChem or the Cambridge Structural Database (CSD)) in the reference list."],"model":"deepseek/deepseek-v4-flash","agent_version":"fair_agent_v8","fulltext_source":"epmc_xml"},"fair_model":"deepseek/deepseek-v4-flash","fair_agent_version":"fair_agent_v8","fair_fulltext_source":"epmc_xml","fair_has_llm":true,"fair_computed_at":"2026-07-20T13:38:28.510233Z","clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}