ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
期刊论文

Assessing the possibility of using large language models in ocular surface diseases

Ling, Qian; Xu, Zi-Song; Zeng, Yan-Mei; Hong, Qi; Qian, Xian-Zhe; Hu, Jin-Yu; Pei, Chong-Gang; Wei, Hong; Zou, Jie; Chen, Cheng; Wang, Xiao-Yu; Chen, Xu; Wu, Zhen-Kai*; Shao, Yi*南昌大学; 上海交通大学
INTERNATIONAL JOURNAL OF OPHTHALMOLOGY2025-1-18DOI: 10.18240/ijo.2025.01.01
1阅读
引用收藏
分享

摘要

AIM: To assess the possibility of using different large language models (LLMs) in ocular surface diseases by selecting five different LLMS to test their accuracy in answering specialized questions related to ocular surface diseases: ChatGPT-4, ChatGPT-3.5, Claude 2, PaLM2, and SenseNova. METHODS: A group of experienced ophthalmology professors were asked to develop a 100-question single- choice question on ocular surface diseases designed to assess the performance of LLMs and human participants in answering ophthalmology specialty exam questions. The exam includes questions on the following topics: keratitis disease (20 questions), keratoconus, keratomalaciac, corneal dystrophy, corneal degeneration, erosive corneal ulcers, and corneal lesions associated with systemic diseases (20 questions), conjunctivitis disease (20 questions), trachoma, pterygoid and conjunctival tumor diseases (20 questions), and dry eye disease (20 questions). Then the total score of each LLMs and compared their mean score, mean correlation, variance, and confidence were calculated. RESULTS: GPT-4 exhibited the highest performance in terms of LLMs. Comparing the average scores of the LLMs group with the four human groups, chief physician, attending physician, regular trainee, and graduate student, it was found that except for ChatGPT-4, the total score of the rest of the LLMs is lower than that of the graduate student group, which had the lowest score in the human group. Both ChatGPT-4 and PaLM2 were more likely to give exact and correct answers, giving very little chance of an incorrect answer. ChatGPT-4 showed higher credibility when answering questions, with a success rate of 59%, but gave the wrong answer to the question 28% of the time. CONCLUSION: GPT-4 model exhibits excellent performance in both answer relevance and confidence. PaLM2 shows a positive correlation (up to 0.8) in terms of answer accuracy during the exam. In terms of answer confidence, PaLM2 is second only to GPT4 and surpasses Claude 2, SenseNova, and GPT-3.5. Despite the fact that ocular surface disease is a highly specialized discipline, GPT-4 still exhibits superior performance, suggesting that its potential and ability to be applied in this field is enormous, perhaps with the potential to be a valuable resource for medical students and clinicians in the future.

学科与关键词

关键词
ChatGPT-4.0ChatGPT-3.5large language modelsocular surface diseases
相关
加载中…
  • Development and evaluation of a large language model of ophthalmology in Chinese
    Zheng, Ce; Ye, Hongfei; Guo, Jinming; Yang, Junrui; Fei, Ping; Yuan, Yuanzhi; Huang, Danqing; Huang, Yuqiang; Peng, Jie; Xie, Xiaoling; Xie, Meng; Zhao, Peiquan; Chen, Li*; Zhang, Mingzhi*British Journal of Ophthalmology, 2024, 108(10): 1390-1397.被引 3
  • ChatGPT and Beyond: An overview of the growing field of large language models and their use in ophthalmology
    Kedia, Nikita; Sanjeev, Suvansh; Ong, Joshua; Chhablani, Jay*Eye, 2024, 38(7): 1252-1261.被引 6
  • ChatGPT and the German board examination for ophthalmology: an evaluation
    Yaici, Remi; Cieplucha, M.; Bock, R.; Moayed, F.; Bechrakis, N. E.; Berens, P.; Feltgen, N.; Friedburg, D.; Graef, M.; Guthoff, R.; Hoffmann, E. M.; Hoerauf, H.; Hintschich, C.; Kohnen, T.; Messmer, E. M.; Nentwich, M. M.; Pleyer, U.; Schaudig, U.; Seitz, B.; Geerling, G.; Roth, M.OPHTHALMOLOGIE, 2024, 121(7): 554-564.
查看全部 →

发表信息

论文状态
公开发表
期刊
INTERNATIONAL JOURNAL OF OPHTHALMOLOGY
发表日期
2025-01-18
卷期
第 18 卷, 第 1 期
页码
1-8
DOI
10.18240/ijo.2025.01.01
成果收录
SCIE
资助标注
National Natural Science Foundation of China [82160195, 82460203]; Degree and Postgraduate Education Teaching Reform Project of Jiangxi Province [JXYJG-2020-026]

作者

  • L

    Ling, Qian (第一作者)

    nanchang univ

  • X

    Xu, Zi-Song

    nanchang univ

查看全部 14 位作者收起作者
  • Z

    Zeng, Yan-Mei

    nanchang univ

  • H

    Hong, Qi

    nanchang univ

  • Q

    Qian, Xian-Zhe

    nanchang univ

  • H

    Hu, Jin-Yu

    nanchang univ

  • P

    Pei, Chong-Gang

    nanchang univ

  • W

    Wei, Hong

    nanchang univ

  • Z

    Zou, Jie

    nanchang univ

  • C

    Chen, Cheng

    nanchang univ

  • W

    Wang, Xiao-Yu

    nanchang univ

  • C

    Chen, Xu

  • W

    Wu, Zhen-Kai (通讯)

  • S

    Shao, Yi (通讯)

    shanghai jiao tong univ

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417