ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
期刊论文

Slit Lamp Report Generation and Question Answering: Development and Validation of a Multimodal Transformer Model with Large Language Model Integration

Zhao, Ziwei; Zhang, Weiyi; Chen, Xiaolan; Song, Fan; Gunasegaram, James; Huang, Wenyong; Shi, Danli; He, Mingguang; Liu, Na*中山大学
JOURNAL OF MEDICAL INTERNET RESEARCH2024-12-30DOI: 10.2196/54047
6阅读
引用收藏
分享

摘要

Background: Large language models have shown remarkable efficacy in various medical research and clinical applications. However, their skills in medical image recognition and subsequent report generation or question answering (QA) remain limited. Objective: We aim to finetune a multimodal, transformer-based model for generating medical reports from slit lamp images and develop a QA system using Llama2. We term this entire process slit lamp-GPT. Methods: Our research used a dataset of 25,051 slit lamp images from 3409 participants, paired with their corresponding physician-created medical reports. We used these data, split into training, validation, and test sets, to finetune the Bootstrapping Language-Image Pre-training framework toward report generation. The generated text reports and human-posed questions were then input into Llama2 for subsequent QA. We evaluated performance using qualitative metrics (including BLEU [bilingual evaluation understudy], CIDEr [consensus-based image description evaluation], ROUGE-L [Recall-Oriented Understudy for Gisting Evaluation-Longest Common Subsequence], SPICE [Semantic Propositional Image Caption Evaluation], accuracy, sensitivity, specificity, precision, and F 1-score) and the subjective assessments of two experienced ophthalmologists on a 1-3 scale (1 referring to high quality). Results: We identified 50 conditions related to diseases or postoperative complications through keyword matching in initial reports. The refined slit lamp-GPT model demonstrated BLEU scores (1-4) of 0.67, 0.66, 0.65, and 0.65, respectively, with a CIDEr score of 3.24, a ROUGE (Recall-Oriented Understudy for Gisting Evaluation) score of 0.61, and a Semantic Propositional Image Caption Evaluation score of 0.37. The most frequently identified conditions were cataracts (22.95%), age-related cataracts (22.03%), and conjunctival concretion (13.13%). Diseaseclassification metricsdemonstratedanoverall accuracy of 0.82 and an F 1-score of 0.64, with high accuracies (>= 0.9) observed for intraocular lens, conjunctivitis, and chronic conjunctivitis, and high F 1-scores (>= 0.9) observed for cataract and age-related cataract. For both report generation and QA components, the two evaluating ophthalmologists reached substantial agreement, with kappa scores between 0.71 and 0.84. In assessing 100 generated reports, they awarded scores of 1.36 for both completeness and correctness; 64% (64/100) wereconsidered "entirely good," and 93% (93/100) were "acceptable." In the evaluation of 300 generated answers to questions, the scores were 1.33 for completeness, 1.14 for correctness, and 1.15 for possible harm, with 66.3% (199/300) rated as "entirely good" and 91.3% (274/300) as "acceptable." Conclusions:This study introduces the slit lamp-GPT model for report generation and subsequent QA, highlighting the potential of large language models to assist ophthalmologists and patients.

学科与关键词

关键词
large language modelslit lampmedical report generationquestion answering
相关
加载中…

文献加载失败,请重试

重试

发表信息

论文状态
公开发表
期刊
JOURNAL OF MEDICAL INTERNET RESEARCH
发表日期
2024-12-30
卷期
第 26 卷
页码
e54047
DOI
10.2196/54047
语言
ENGLISH
成果收录
SCIE
资助标注
Global STEM Professorship Scheme [P0046113]; Henry G. Leong Endowed Professorship in Elderly Vision Health

作者

  • Z

    Zhao, Ziwei (第一作者)

    hong kong polytech univ

  • Z

    Zhang, Weiyi

    hong kong polytech univ

查看全部 9 位作者收起作者
  • C

    Chen, Xiaolan

    hong kong polytech univ

  • S

    Song, Fan

    hong kong polytech univ

  • G

    Gunasegaram, James

  • H

    Huang, Wenyong

    sun yat sen univ

  • S

    Shi, Danli

    hong kong polytech univ

  • H

    He, Mingguang

    hong kong polytech univ

  • L

    Liu, Na (通讯)

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417