{"doi":"10.1016/j.patter.2024.100933","title":"TCGA-Reports: A machine-readable pathology report resource for benchmarking text-based AI models","abstract":"In cancer research, pathology report text is a largely untapped data source. Pathology reports are routinely generated, more nuanced than structured data, and contain added insight from pathologists. However, there are no publicly available datasets for benchmarking report-based models. Two recent advances suggest the urgent need for a benchmark dataset. First, improved optical character recognition (OCR) techniques will make it possible to access older pathology reports in an automated way, increasing the data available for analysis. Second, recent improvements in natural language processing (NLP) techniques using artificial intelligence (AI) allow more accurate prediction of clinical targets from text. We apply state-of-the-art OCR and customized post-processing to report PDFs from The Cancer Genome Atlas, generating a machine-readable corpus of 9,523 reports. Finally, we perform a proof-of-principle cancer-type classification across 32 tissues, achieving 0.992 average AU-ROC. This dataset will be useful to researchers across specialties, including research clinicians, clinical trial investigators, and clinical NLP researchers.","journal":"Patterns","year":2024,"id":420029,"datarank":1.3952836492354932,"base_score":3.6109179126442243,"endowment":3.6109179126442243,"self_citation_contribution":0.5416376868966337,"citation_network_contribution":0.8536459623388595,"self_endowment_contribution":0.5416376868966337,"citer_contribution":0.8536459623388595,"corpus_percentile":85.2092519532761,"corpus_rank":1913,"citation_count":36,"citer_count":34,"citers_with_citation_signal":20,"citers_with_endowment":20,"datacite_reuse_total":0,"is_dataset":true,"is_dataset_confidence":0.9145,"is_data_producer":false,"deposit_databanks":null,"is_oa":true,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":"2024-01-01","fair_score":70.8333,"fair_percentile":91.99021705900336,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":2011,"name":"Nicholas P. Tatonetti","orcid":"0000-0002-2700-2597","position":1,"is_corresponding":false},{"id":1159465,"name":"Jenna Kefeli","orcid":null,"position":0,"is_corresponding":true}],"reference_count":47,"raw_metadata":null,"created_at":"2026-07-19T01:57:25.777459Z","pmid":"38487800","pmcid":"PMC10935496","fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":null,"license":null,"views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":66.6667,"fair_a":75.0,"fair_i":20.0,"fair_r":41.6667,"fair_zscore":1.4401,"fair_rationale":{"fair_score":70.83,"has_llm":true,"taxonomy_version":"fair_taxonomy_v5","dimensions":{"F":{"name":"Findable","score":66.67,"criteria":[{"key":"f_dataset_pid","label":"Persistent identifier for the data","kind":"llm","weight":2.0,"fraction":1.0,"verdict":"yes","evidence":"10.17632/hyg5xkznpx.1","grounded":true,"rationale":"The dataset has a DOI (10.17632/hyg5xkznpx.1) in the reference list, a persistent identifier scheme. [majority verdict 'yes' (4/5 passes agreed)]","anchors":["RDA-F1-01D — FAIR Data Maturity Model: 'Data is identified by a persistent identifier' (priorit","RDA-F1-02D — FAIR Data Maturity Model: 'Data is identified by a globally unique identifier'","FsF-F1-02D — F-UJI/FAIRsFAIR: 'Data is assigned a persistent identifier'"],"scored":true,"signal":null},{"key":"f_repository_named","label":"Named repository","kind":"llm","weight":2.0,"fraction":1.0,"verdict":"yes","evidence":"Text reports are available on Mendeley.","grounded":true,"rationale":"Mendeley Data is a named repository, meeting the yes definition. [majority verdict 'yes' (4/5 passes agreed)]","anchors":["RDA-F4-01M — FAIR Data Maturity Model: metadata is offered so it can be harvested and indexed (","NIH DMS Policy Element 4 (NOT-OD-21-014) — name the repository where data will be archived","NSTC Desirable Characteristics of Data Repositories (2022) — 'Long-Term Sustainability', 'Reten"],"scored":true,"signal":null},{"key":"f_data_availability_statement","label":"Data-availability statement","kind":"llm","weight":2.0,"fraction":0.5,"verdict":"partial","evidence":"Text reports are available on Mendeley.","grounded":true,"rationale":"The statement names the repository (Mendeley) but does not include the DOI or accession, so it is partial. [majority verdict 'partial' (3/5 passes agreed)]","anchors":["Colavizza, Hrynaszkiewicz, Staden, Whitaker & McGillivray (2020), 'The citation advantage of li","Springer Nature research data policy — Data Availability Statements: standard statement templat","RDA-F3-01M — metadata clearly and explicitly includes the identifier of the data it describes"],"scored":false,"signal":null},{"key":"f_discovery_metadata","label":"Description of the dataset as an object","kind":"llm","weight":2.0,"fraction":0.0,"verdict":"no","evidence":"The final dataset consisted of 9,523 reports (23,909 pages, or 842,134 lines) across 32 cancer types.","grounded":false,"rationale":"The dataset content is described in running prose only, without an itemised inventory or table of files. [downgraded to 'no' — no verifiable quote from the paper] [majority verdict 'no' (2/5 passes agreed)]","anchors":["RDA-F2-01M — 'Rich metadata is provided to allow discovery' (priority Essential)","FsF-F2-01M — F-UJI: 'Metadata includes descriptive core elements to support data findability'","FsF-R1-01MD — F-UJI: 'Metadata specifies the content of the data'"],"scored":false,"signal":null},{"key":"f_dataset_cited","label":"Dataset formally cited","kind":"llm","weight":1.0,"fraction":1.0,"verdict":"yes","evidence":"Kefeli J. Tatonetti N. TCGA-Reports: A Machine-Readable Pathology Report Resource for Benchmarking Text-Based AI Models Mendeley Data 2024 10.17632/hyg5xkznpx.1 PMC10935496 38487800","grounded":true,"rationale":"The dataset identifier (DOI) appears as a reference-list entry, which qualifies as a citation. [majority verdict 'yes' (3/5 passes agreed)]","anchors":["FORCE11 Joint Declaration of Data Citation Principles (2014) — data should be cited as a first-","RDA-F3-01M — metadata clearly and explicitly includes the identifier of the data it describes","FsF-F3-01M — F-UJI: 'Metadata includes the identifier of the data it describes'"],"scored":true,"signal":null}]},"A":{"name":"Accessible","score":75.0,"criteria":[{"key":"a_data_openly_accessible","label":"Access route free of preconditions","kind":"llm","weight":2.0,"fraction":1.0,"verdict":"yes","evidence":"We make the final corpus of 9,523 patient reports publicly available for researchers to use for data mining or machine learning applications.","grounded":true,"rationale":"The paper states the data are publicly available on Mendeley with no stated precondition. [majority verdict 'yes' (3/5 passes agreed)]","anchors":["RDA-A1.1-01D — 'Data is accessible through a free access protocol'","FsF-A1-01M — F-UJI: 'Metadata contains access level and access conditions of the data'","NSTC Desirable Characteristics of Data Repositories (2022) — 'Free and Easy Access'"],"scored":true,"signal":null},{"key":"a_access_conditions_stated","label":"Access level labelled","kind":"llm","weight":1.0,"fraction":1.0,"verdict":"yes","evidence":"We make the final corpus of 9,523 patient reports publicly available for researchers to use for data mining or machine learning applications.","grounded":true,"rationale":"The paper explicitly labels the dataset as 'publicly available' in the introduction, meeting the yes definition. [majority verdict 'yes' (4/5 passes agreed)]","anchors":["FsF-A1-01M — F-UJI: 'Metadata contains access level and access conditions of the data'","RDA-A1-01M — metadata contains information to enable the user to get access to the data","COAR Controlled Vocabularies — Access Rights v1.0 (open / embargoed / restricted / metadata-onl"],"scored":false,"signal":null},{"key":"a_controlled_access_for_sensitive","label":"Gatekeeper for sensitive data","kind":"llm","weight":0.5,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"The data are de-identified and publicly available with no gatekeeper named; no access control is mentioned.","anchors":["NIH Genomic Data Sharing Policy (NOT-OD-14-124) — controlled-access via a Data Access Committee","RDA-A1.2-01D — 'Data is accessible through an access protocol that supports authentication and ","NIH DMS Policy Element 5 (NOT-OD-21-014) — Access, Distribution, or Reuse Considerations (conse"],"scored":false,"signal":null},{"key":"a_timeline_retention","label":"Availability timing & retention","kind":"llm","weight":0.5,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No sentence states when the data become available or how long they persist. [majority verdict 'no' (4/5 passes agreed)]","anchors":["NIH DMS Plan Element 4 (NOT-OD-21-014) — Data Preservation, Access, and Associated Timelines","NSTC Desirable Characteristics (2022), Organizational Infrastructure: 'Retention Policy'","RDA-A2-01M — 'Metadata is guaranteed to remain available after data is no longer available'"],"scored":false,"signal":null}]},"I":{"name":"Interoperable","score":20.0,"criteria":[{"key":"i_open_nonproprietary_format","label":"Open file format","kind":"llm","weight":1.0,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No file format is named for the released data. [majority verdict 'no' (4/5 passes agreed)]","anchors":["FsF-R1.3-02D — F-UJI: 'Data is available in a file format recommended by the target research co","RDA-R1.3-02D — data is expressed in a machine-understandable community standard","RDA-I1-01D — data uses a knowledge representation expressed in a standardised format"],"scored":true,"signal":null},{"key":"i_community_standard_vocabulary","label":"Community standard / vocabulary","kind":"llm","weight":1.0,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No data or metadata community standard is named in the paper.","anchors":["RDA-R1.3-01M — 'Metadata complies with a community standard' (priority Essential)","RDA-R1.3-01D — 'Data complies with a community standard'","RDA-I2-01M — '(Meta)data use vocabularies that follow FAIR principles'"],"scored":false,"signal":null},{"key":"i_qualified_references","label":"Identifiers for the resources the data depend on","kind":"llm","weight":0.5,"fraction":1.0,"verdict":"yes","evidence":"10.1038/sdata.2016.35","grounded":true,"rationale":"The paper references the MIMIC-III database with a DOI, an identifier for a resource other than its own dataset. [majority verdict 'yes' (4/5 passes agreed)]","anchors":["RDA-I3-01M — '(meta)data include references to other (meta)data'","RDA-I3-03M — 'metadata includes qualified references to other metadata'","FsF-I3-01M — F-UJI: 'Metadata includes links between the data and its related entities'"],"scored":false,"signal":null}]},"R":{"name":"Reusable","score":41.67,"criteria":[{"key":"r_reuse_license","label":"Reuse licence","kind":"llm","weight":2.0,"fraction":0.0,"verdict":"no","evidence":null,"grounded":false,"rationale":"No licence is stated for the data; the CC BY license covers only the article.","anchors":["RDA-R1.1-01M — 'Metadata includes information about the licence under which the data can be reu","RDA-R1.1-02M — 'Metadata refers to a standard reuse licence'","RDA-R1.1-03M — 'Metadata refers to a machine-understandable reuse licence'"],"scored":true,"signal":null},{"key":"r_provenance_methods","label":"Provenance of the data","kind":"llm","weight":1.0,"fraction":0.5,"verdict":"partial","evidence":"We used OCR to transform reports from PDF into text. We qualitatively evaluated several different OCR programs, with Textract producing the most accurate results.","grounded":false,"rationale":"The paper names specific tools (Textract, pytesseract, ClinicalBERT) used to produce the data. [downgraded to 'partial' — no verifiable quote from the paper] [majority verdict 'partial' (4/5 passes agreed)]","anchors":["RDA-R1.2-01M — 'Metadata includes provenance information according to community- specific standa","FsF-R1.2-01M — F-UJI: 'Metadata includes provenance information about data creation or generati","W3C PROV-O (W3C Recommendation, 2013) — the entity/activity/agent model of provenance"],"scored":false,"signal":null},{"key":"r_documentation_codebook","label":"Documentation / codebook","kind":"llm","weight":1.0,"fraction":0.5,"verdict":"partial","evidence":"Table 1 Demographic characteristics of patients in final pathology report dataset","grounded":true,"rationale":"Variable definitions (demographics) are provided inside the article in Table 1, but no documentation object is said to accompany the data. [majority verdict 'partial' (4/5 passes agreed)]","anchors":["RDA-R1-01M — '(Meta)data are richly described with a plurality of accurate and relevant attribu","FsF-R1-01MD — F-UJI: 'Metadata specifies the content of the data'","NIH DMS Policy Element 3 (NOT-OD-21-014) — Standards (documentation and metadata to accompany t"],"scored":false,"signal":null},{"key":"r_versioning","label":"Snapshot identified","kind":"llm","weight":0.5,"fraction":0.0,"verdict":"no","evidence":"a_access_conditions_stated","grounded":false,"rationale":"No version token or date is stated for the data snapshot. [majority verdict 'no' (3/5 passes agreed)]","anchors":["DataCite Metadata Schema 4.6 — the 'Version' property","RDA-R1.2-01M — provenance information (which version was used is provenance)","NSTC Desirable Characteristics of Data Repositories (2022) — 'Provenance', 'Retention Policy'"],"scored":true,"signal":null},{"key":"x_code_availability","label":"Analysis code available","kind":"llm","weight":1.0,"fraction":1.0,"verdict":"yes","evidence":"https://github.com/jkefeli/tcga-path-reports","grounded":true,"rationale":"The paper gives a GitHub repository URL for the code. [majority verdict 'yes' (3/5 passes agreed)]","anchors":["NIH DMS Policy Element 2 (NOT-OD-21-014) — 'Related Tools, Software and/or Code'","FAIR4RS Principles v1.0 (Chue Hong et al., 2022; RDA/FORCE11/ReSA) — FAIR Principles for Resear","FORCE11 Software Citation Principles (Smith, Katz & Niemeyer, 2016, PeerJ CS 2:e86)"],"scored":true,"signal":null},{"key":"x_funding_attribution","label":"Funder and award number","kind":"llm","weight":0.5,"fraction":1.0,"verdict":"yes","evidence":"R35GM131905","grounded":true,"rationale":"An alphanumeric grant number (R35GM131905) is given for the NIH NIGMS funding. [majority verdict 'yes' (4/5 passes agreed)]","anchors":["DataCite Metadata Schema 4.6 — 'FundingReference' property (funderName, funderIdentifier, award","Crossref Funder Registry — canonical funder identifiers for funding metadata","RDA-F2-01M — rich metadata provided to allow discovery (funding is part of the descriptive reco"],"scored":true,"signal":null}]}},"actions":[{"key":"r_reuse_license","dimension":"R","label":"Reuse licence","action":"Attach a standard, machine-readable open licence to the deposit — CC0 or CC BY, which is what Horizon Europe and most funders expect — and print the licence identifier in the paper. 'Free to use' is not a licence: it grants nothing a reuser's institution can rely on.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No licence is stated for the data; the CC BY license covers only the article.","gain":16.67,"priority":"essential","scored":true},{"key":"i_open_nonproprietary_format","dimension":"I","label":"Open file format","action":"Release the data in an open, community-standard format (CSV/TSV, JSON, HDF5, NetCDF, FASTQ, VCF, NIfTI…) instead of — or alongside — any proprietary or instrument-native format, and name the format in the paper. A dataset that needs a €2,000 licence to open is not reusable.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No file format is named for the released data. [majority verdict 'no' (4/5 passes agreed)]","gain":8.33,"priority":"important","scored":true},{"key":"r_versioning","dimension":"R","label":"Snapshot identified","action":"Version the deposit and cite the exact version analysed (a version-specific DOI, or an accession with its version suffix). A reader reproducing your work against 'the current release' is reproducing it against a different dataset.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":"a_access_conditions_stated","why":"No version token or date is stated for the data snapshot. [majority verdict 'no' (3/5 passes agreed)]","gain":4.17,"priority":"useful","scored":true},{"key":"f_data_availability_statement","dimension":"F","label":"Data-availability statement","action":"Replace the statement with the repository template: name the repository and give the accession or DOI (Colavizza category 3). This is the only DAS class associated with a measured citation advantage; 'available on reasonable request' and 'within the article' are not.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"Text reports are available on Mendeley.","why":"The statement names the repository (Mendeley) but does not include the DOI or accession, so it is partial. [majority verdict 'partial' (3/5 passes agreed)]","gain":0.0,"priority":"essential","scored":false},{"key":"f_discovery_metadata","dimension":"F","label":"Description of the dataset as an object","action":"Add a 'Data Records' section: itemise every file in the deposit and every variable or sample it holds, with counts and units. Describe the dataset as an object in its own right, not as a by-product of the findings — this is what makes it discoverable to someone who is not looking for your paper.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":"The final dataset consisted of 9,523 reports (23,909 pages, or 842,134 lines) across 32 cancer types.","why":"The dataset content is described in running prose only, without an itemised inventory or table of files. [downgraded to 'no' — no verifiable quote from the paper] [majority verdict 'no' (2/5 passes agreed)]","gain":0.0,"priority":"essential","scored":false},{"key":"i_community_standard_vocabulary","dimension":"I","label":"Community standard / vocabulary","action":"Adopt and NAME your domain's data standard — the minimum-information checklist, metadata schema, or ontology your community uses (MIAME/MINSEQE, ISA-Tab, BIDS, an OBO ontology, HL7 FHIR/OMOP) — and say which one you followed. A reporting checklist standardises your paper; it does nothing for your data. In clinical / human-subjects, describe the data with OMOP CDM, CDISC SDTM or HL7 FHIR.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No data or metadata community standard is named in the paper.","gain":0.0,"priority":"important","scored":false},{"key":"r_provenance_methods","dimension":"R","label":"Provenance of the data","action":"Name the instruments, kits, and software — with versions — that produced the data, not just the verbs. 'Reads were aligned' is not provenance; 'aligned with STAR v2.7.9a to GRCh38' is, because someone else can rerun it.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"We used OCR to transform reports from PDF into text. We qualitatively evaluated several different OCR programs, with Textract producing the most accurate results.","why":"The paper names specific tools (Textract, pytesseract, ClinicalBERT) used to produce the data. [downgraded to 'partial' — no verifiable quote from the paper] [majority verdict 'partial' (4/5 passes agreed)]","gain":0.0,"priority":"important","scored":false},{"key":"r_documentation_codebook","dimension":"R","label":"Documentation / codebook","action":"Ship a README and a data dictionary IN the deposit — every file, every variable, its units, its allowed values, its missing-value codes. It is the cheapest single thing that makes a dataset usable by someone who was not in the lab, and a table buried in the article does not travel with the data.","anchors":["yes","partial","no"],"verdict":"partial","current":0.5,"evidence":"Table 1 Demographic characteristics of patients in final pathology report dataset","why":"Variable definitions (demographics) are provided inside the article in Table 1, but no documentation object is said to accompany the data. [majority verdict 'partial' (4/5 passes agreed)]","gain":0.0,"priority":"important","scored":false},{"key":"a_controlled_access_for_sensitive","dimension":"A","label":"Gatekeeper for sensitive data","action":"Route sensitive data through an institutional gatekeeper — deposit in a controlled- access repository (dbGaP, EGA) with a Data Access Committee and a published DUA — rather than through the corresponding author's inbox. An author-gated dataset dies with the author's email address, and 'on reasonable request' has been shown repeatedly not to yield data. For sensitive/human clinical / human-subjects data, use a controlled-access repository such as dbGaP or EGA.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"The data are de-identified and publicly available with no gatekeeper named; no access control is mentioned.","gain":0.0,"priority":"useful","scored":false},{"key":"a_timeline_retention","dimension":"A","label":"Availability timing & retention","action":"State when the data become available AND how long they will be retained — cite the repository's preservation policy. NIH DMS Element 4 asks for both; most papers give neither.","anchors":["yes","partial","no"],"verdict":"no","current":0.0,"evidence":null,"why":"No sentence states when the data become available or how long they persist. [majority verdict 'no' (4/5 passes agreed)]","gain":0.0,"priority":"useful","scored":false}],"suggestions":["Attach a standard, machine-readable open licence to the deposit — CC0 or CC BY, which is what Horizon Europe and most funders expect — and print the licence identifier in the paper. 'Free to use' is not a licence: it grants nothing a reuser's institution can rely on.","Release the data in an open, community-standard format (CSV/TSV, JSON, HDF5, NetCDF, FASTQ, VCF, NIfTI…) instead of — or alongside — any proprietary or instrument-native format, and name the format in the paper. A dataset that needs a €2,000 licence to open is not reusable.","Version the deposit and cite the exact version analysed (a version-specific DOI, or an accession with its version suffix). A reader reproducing your work against 'the current release' is reproducing it against a different dataset.","Replace the statement with the repository template: name the repository and give the accession or DOI (Colavizza category 3). This is the only DAS class associated with a measured citation advantage; 'available on reasonable request' and 'within the article' are not.","Add a 'Data Records' section: itemise every file in the deposit and every variable or sample it holds, with counts and units. Describe the dataset as an object in its own right, not as a by-product of the findings — this is what makes it discoverable to someone who is not looking for your paper."],"model":"deepseek/deepseek-v4-flash","agent_version":"fair_agent_v8","fulltext_source":"epmc_xml"},"fair_model":"deepseek/deepseek-v4-flash","fair_agent_version":"fair_agent_v8","fair_fulltext_source":"epmc_xml","fair_has_llm":true,"fair_computed_at":"2026-07-20T11:29:45.014054Z","clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}