{"doi":"10.1093/jamia/ocae117","title":"Evaluation of GPT-4 ability to identify and generate patient instructions for actionable incidental radiology findings","abstract":"OBJECTIVES: To evaluate the proficiency of a HIPAA-compliant version of GPT-4 in identifying actionable, incidental findings from unstructured radiology reports of Emergency Department patients. To assess appropriateness of artificial intelligence (AI)-generated, patient-facing summaries of these findings. MATERIALS AND METHODS: Radiology reports extracted from the electronic health record of a large academic medical center were manually reviewed to identify non-emergent, incidental findings with high likelihood of requiring follow-up, further sub-stratified as \"definitely actionable\" (DA) or \"possibly actionable-clinical correlation\" (PA-CC). Instruction prompts to GPT-4 were developed and iteratively optimized using a validation set of 50 reports. The optimized prompt was then applied to a test set of 430 unseen reports. GPT-4 performance was primarily graded on accuracy identifying either DA or PA-CC findings, then secondarily for DA findings alone. Outputs were reviewed for hallucinations. AI-generated patient-facing summaries were assessed for appropriateness via Likert scale. RESULTS: For the primary outcome (DA or PA-CC), GPT-4 achieved 99.3% recall, 73.6% precision, and 84.5% F-1. For the secondary outcome (DA only), GPT-4 demonstrated 95.2% recall, 77.3% precision, and 85.3% F-1. No findings were \"hallucinated\" outright. However, 2.8% of cases included generated text about recommendations that were inferred without specific reference. The majority of True Positive AI-generated summaries required no or minor revision. CONCLUSION: GPT-4 demonstrates proficiency in detecting actionable, incidental findings after refined instruction prompting. AI-generated patient instructions were most often appropriate, but rarely included inferred recommendations. While this technology shows promise to augment diagnostics, active clinician oversight via \"human-in-the-loop\" workflows remains critical for clinical implementation.","journal":"Journal of the American Medical Informatics Association","year":2024,"id":429895,"datarank":0.47032413238937254,"base_score":3.1354942159291497,"endowment":3.1354942159291497,"self_citation_contribution":0.47032413238937254,"citation_network_contribution":0.0,"self_endowment_contribution":0.47032413238937254,"citer_contribution":0.0,"corpus_percentile":null,"corpus_rank":null,"citation_count":22,"citer_count":0,"citers_with_citation_signal":0,"citers_with_endowment":0,"datacite_reuse_total":0,"is_dataset":false,"is_dataset_confidence":0.5482,"is_data_producer":false,"deposit_databanks":null,"is_oa":true,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":"2024-01-01","fair_score":null,"fair_percentile":null,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":1232919,"name":"Gregory W Simon","orcid":null,"position":1,"is_corresponding":false},{"id":1232920,"name":"Olumide Akindutire","orcid":null,"position":2,"is_corresponding":false},{"id":52496,"name":"Yindalon Aphinyanaphongs","orcid":"0000-0001-8605-5392","position":3,"is_corresponding":false},{"id":304309,"name":"Jonathan Austrian","orcid":"0000-0002-0016-4705","position":4,"is_corresponding":false},{"id":1051930,"name":"Jung G. Kim","orcid":"0000-0002-0778-4931","position":5,"is_corresponding":false},{"id":704505,"name":"Nicholas Genes","orcid":"0000-0002-9836-2477","position":6,"is_corresponding":false},{"id":1232921,"name":"Jacob A Goldenring","orcid":null,"position":7,"is_corresponding":false},{"id":311820,"name":"Vincent J. Major","orcid":"0000-0003-2604-3458","position":8,"is_corresponding":false},{"id":1232922,"name":"Chloé S Pariente","orcid":null,"position":9,"is_corresponding":false},{"id":1232923,"name":"Edwin G Pineda","orcid":null,"position":10,"is_corresponding":false},{"id":281362,"name":"Stella K. Kang","orcid":"0000-0003-2402-3787","position":11,"is_corresponding":false},{"id":1232560,"name":"Kar-mun Woo","orcid":"0009-0004-6117-5303","position":0,"is_corresponding":true}],"reference_count":24,"raw_metadata":{"citation_network_status":"fetched"},"created_at":"2026-07-19T01:59:11.328802Z","pmid":"38778578","pmcid":null,"fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":null,"license":null,"views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":null,"fair_a":null,"fair_i":null,"fair_r":null,"fair_zscore":null,"fair_rationale":null,"fair_model":null,"fair_agent_version":null,"fair_fulltext_source":null,"fair_has_llm":null,"fair_computed_at":null,"clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}