ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
期刊论文

OphGLM: An ophthalmology large language-and-vision assistant

Deng, Zhuo; Gao, Weihao; Chen, Chucheng; Niu, Zhiyuan; Gong, Zheng; Zhang, Ruiheng; Cao, Zhenjie; Li, Fang; Ma, Zhaoyi; Wei, Wenbin; Ma, Lan*清华大学
Artificial Intelligence in Medicine2024-11DOI: 10.1016/j.artmed.2024.103001
1阅读
引用收藏
分享

摘要

Vision computer-aided diagnostic methods have been used in early ophthalmic disease screening and diagnosis. However, the limited output formats of these methods lead to poor human-computer interaction and low clinical applicability value. Thus, ophthalmic visual question answering is worth studying. Unfortunately, no practical solutions exist before Large Language Models(LLMs). In this paper, we investigate the ophthalmic visual diagnostic interaction problem. We construct an ophthalmology large language-and-vision assistant, OphGLM, consisting of an image encoder, a text encoder, a fusion module, and an LLM module. We establish anew Chinese ophthalmic fine-tuning dataset, FundusTuning-CN, including the fundus instruction and conversation sets. Based on FundusTuning-CN, we establish a novel LLM-tuning strategy to introduce visual model understanding and ophthalmic knowledge into LLMs at a low cost and high efficiency. Leveraging the pre-training of the image encoder, OphGLM demonstrates strong visual understanding and surpasses opensource visual language models in common fundus disease classification tasks. The FundusTuning-CN enables OphGLM to surpass open-source medical LLMs in both ophthalmic knowledge and interactive capabilities. Our proposed OphGLM has the potential to revolutionize clinical applications in ophthalmology. The dataset, code, and models will be publicly available at https://github.com/ML-AILab/OphGLM.

学科与关键词

关键词
OphthalmologyVisual dialogue interactionLarge language models
相关
加载中…
  • Development and evaluation of a large language model of ophthalmology in Chinese
    Zheng, Ce; Ye, Hongfei; Guo, Jinming; Yang, Junrui; Fei, Ping; Yuan, Yuanzhi; Huang, Danqing; Huang, Yuqiang; Peng, Jie; Xie, Xiaoling; Xie, Meng; Zhao, Peiquan; Chen, Li*; Zhang, Mingzhi*British Journal of Ophthalmology, 2024, 108(10): 1390-1397.被引 3
  • Slit Lamp Report Generation and Question Answering: Development and Validation of a Multimodal Transformer Model with Large Language Model Integration
    Zhao, Ziwei; Zhang, Weiyi; Chen, Xiaolan; Song, Fan; Gunasegaram, James; Huang, Wenyong; Shi, Danli; He, Mingguang; Liu, Na*JOURNAL OF MEDICAL INTERNET RESEARCH, 2024, 26: e54047.
  • ChatGPT and Beyond: An overview of the growing field of large language models and their use in ophthalmology
    Kedia, Nikita; Sanjeev, Suvansh; Ong, Joshua; Chhablani, Jay*Eye, 2024, 38(7): 1252-1261.被引 6
查看全部 →

发表信息

论文状态
公开发表
期刊
Artificial Intelligence in Medicine
发表日期
2024-11
卷期
第 157 卷
页码
103001
DOI
10.1016/j.artmed.2024.103001
语言
ENGLISH
成果收录
SCIE
资助标注
Science and Technology Innovation Committee of Shenzhen-Platform and Carrier, China (International Science and Technology Information Center); Shenzhen Bay Lab, China; Shenzhen Science and Technology Innovation Committee [KCXFZ20211020163813019, JCYJ20230807094803007]; National Natural Science Foundation of China [82220108017]

作者

  • D

    Deng, Zhuo (第一作者)

    tsinghua univ

  • G

    Gao, Weihao

    tsinghua univ

查看全部 11 位作者收起作者
  • C

    Chen, Chucheng

    tsinghua univ

  • N

    Niu, Zhiyuan

    tsinghua univ

  • G

    Gong, Zheng

    tsinghua univ

  • Z

    Zhang, Ruiheng

  • C

    Cao, Zhenjie

    tsinghua univ

  • L

    Li, Fang

    tsinghua univ

  • M

    Ma, Zhaoyi

  • W

    Wei, Wenbin

  • M

    Ma, Lan (通讯)

    tsinghua univ

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417