{"doi":"10.1016/j.mcpdig.2024.01.003","title":"Appropriateness of Ophthalmology Recommendations From an Online Chat-Based Artificial Intelligence Model","abstract":"Objective: To determine the appropriateness of ophthalmology recommendations from an online chat-based artificial intelligence model to ophthalmology questions. Patients and Methods: Cross-sectional qualitative study from April 1, 2023, to April 30, 2023. A total of 192 questions were generated spanning all ophthalmic subspecialties. Each question was posed to a large language model (LLM) 3 times. The responses were graded by appropriate subspecialists as appropriate, inappropriate, or unreliable in 2 grading contexts. The first grading context was if the information was presented on a patient information site. The second was an LLM-generated draft response to patient queries sent by the electronic medical record (EMR). Appropriate was defined as accurate and specific enough to serve as a surrogate for physician-approved information. Main outcome measure was percentage of appropriate responses per subspecialty. Results: For patient information site-related questions, the LLM provided an overall average of 79% appropriate responses. Variable rates of average appropriateness were observed across ophthalmic subspecialties for patient information site information ranging from 56% to 100%: cataract or refractive (92%), cornea (56%), glaucoma (72%), neuro-ophthalmology (67%), oculoplastic or orbital surgery (80%), ocular oncology (100%), pediatrics (89%), vitreoretinal diseases (86%), and uveitis (65%). For draft responses to patient questions via EMR, the LLM provided an overall average of 74% appropriate responses and varied by subspecialty: cataract or refractive (85%), cornea (54%), glaucoma (77%), neuro-ophthalmology (63%), oculoplastic or orbital surgery (62%), ocular oncology (90%), pediatrics (94%), vitreoretinal diseases (88%), and uveitis (55%). Stratifying grades across health information categories (disease and condition, risk and prevention, surgery-related, and treatment and management) showed notable but insignificant variations, with disease and condition often rated highest (72% and 69%) for appropriateness and surgery-related (55% and 51%) lowest, in both contexts. Conclusion: This LLM reported mostly appropriate responses across multiple ophthalmology subspecialties in the context of both patient information sites and EMR-related responses to patient questions. Current LLM offerings require optimization and improvement before widespread clinical use.","journal":"Mayo Clinic Proceedings Digital Health","year":2024,"id":430336,"datarank":0.0,"base_score":0.0,"endowment":0.0,"self_citation_contribution":0.0,"citation_network_contribution":0.0,"self_endowment_contribution":0.0,"citer_contribution":0.0,"corpus_percentile":null,"corpus_rank":null,"citation_count":19,"citer_count":0,"citers_with_citation_signal":0,"citers_with_endowment":0,"datacite_reuse_total":0,"is_dataset":false,"is_dataset_confidence":0.7559,"is_data_producer":false,"deposit_databanks":null,"is_oa":true,"file_count":0,"downloads":0,"has_version_chain":false,"published_date":"2024-01-01","fair_score":null,"fair_percentile":null,"algorithm_id":"datarank_citation_only_1hop_v6","ranking_scope":"data_only","authors":[{"id":478523,"name":"Timothy T. Xu","orcid":"0000-0003-4781-985X","position":1,"is_corresponding":false},{"id":523944,"name":"Blake H. Fortes","orcid":"0000-0002-3206-0062","position":2,"is_corresponding":false},{"id":937854,"name":"Raymond Iezzi","orcid":"0000-0002-5126-1898","position":3,"is_corresponding":false},{"id":647626,"name":"Timothy W. Olsen","orcid":"0000-0002-2200-266X","position":4,"is_corresponding":false},{"id":937852,"name":"Matthew R. Starr","orcid":"0000-0002-3021-5630","position":5,"is_corresponding":false},{"id":732399,"name":"Sophie J. Bakri","orcid":"0000-0002-0887-3194","position":6,"is_corresponding":false},{"id":288235,"name":"Brittni A. Scruggs","orcid":"0000-0001-8806-1438","position":7,"is_corresponding":false},{"id":305646,"name":"Andrew J. Barkmeier","orcid":"0000-0002-2666-3828","position":8,"is_corresponding":false},{"id":1233316,"name":"Sanjay V. Patel","orcid":"0000-0001-6694-1530","position":9,"is_corresponding":false},{"id":1233317,"name":"Keith H. Baratz","orcid":"0000-0002-0676-7646","position":10,"is_corresponding":false},{"id":1233318,"name":"Ashlie Bernhisel","orcid":"0000-0002-6692-2281","position":11,"is_corresponding":false},{"id":1233319,"name":"Lilly H. Wagner","orcid":"0000-0003-2970-8339","position":12,"is_corresponding":false},{"id":486610,"name":"Andrea A. Tooley","orcid":"0000-0001-5818-0073","position":13,"is_corresponding":false},{"id":1233320,"name":"Gavin W. Roddy","orcid":"0000-0002-8905-8588","position":14,"is_corresponding":false},{"id":437437,"name":"Arthur J. Sit","orcid":"0000-0002-3904-2625","position":15,"is_corresponding":false},{"id":1233702,"name":"Kristi Y. Wu","orcid":null,"position":16,"is_corresponding":false},{"id":402770,"name":"Erick D. Bothun","orcid":"0000-0002-6749-4768","position":17,"is_corresponding":false},{"id":518028,"name":"Sasha A. Mansukhani","orcid":"0000-0002-7122-4138","position":18,"is_corresponding":false},{"id":383591,"name":"Brian G. Mohney","orcid":"0000-0001-9309-1228","position":19,"is_corresponding":false},{"id":286579,"name":"John J. Chen","orcid":"0000-0003-3644-7528","position":20,"is_corresponding":false},{"id":880439,"name":"Michael C. Brodsky","orcid":null,"position":21,"is_corresponding":false},{"id":1025552,"name":"Deena Tajfirouz","orcid":"0000-0001-8419-8728","position":22,"is_corresponding":false},{"id":1025551,"name":"Kevin D. Chodnicki","orcid":"0000-0002-6815-8272","position":23,"is_corresponding":false},{"id":478525,"name":"Wendy M. Smith","orcid":"0000-0001-9830-9601","position":24,"is_corresponding":false},{"id":889036,"name":"Lauren A. Dalvin","orcid":"0000-0001-9710-4284","position":25,"is_corresponding":false},{"id":1094227,"name":"Prashant D. Tailor","orcid":"0000-0002-6616-3270","position":0,"is_corresponding":true}],"reference_count":11,"raw_metadata":null,"created_at":"2026-07-19T01:59:16.092672Z","pmid":"38577703","pmcid":null,"fwci":null,"citation_percentile":null,"influential_citations":0,"oa_status":null,"license":null,"views":0,"total_file_size_bytes":0,"version_count":0,"fair_f":null,"fair_a":null,"fair_i":null,"fair_r":null,"fair_zscore":null,"fair_rationale":null,"fair_model":null,"fair_agent_version":null,"fair_fulltext_source":null,"fair_has_llm":null,"fair_computed_at":null,"clinical_trials":[],"software_tools":[],"db_accessions":[],"linked_datasets":[],"topics":[]}