摘要
By integrating visual and linguistic understanding, visual-language foundation models (VLFMs) have the great potential to advance the interpretation of medical data, thereby enhancing diagnostic precision, treatment planning, and patient management. We reviewed the developmental strategies of VLFMs, detailing the pretraining strategies, and subsequent application across various healthcare facets. The challenges inherent to VLFMs are described, including safeguarding data privacy amidst sensitive medical data usage, ensuring algorithmic transparency, and fostering explainability for trust in clinical decision-making. We underscored the significance of VLFMs in addressing the complexity of multimodal medical data, from visual to textual, and their potential in tasks like image-based disease diagnosis, medicine report synthesis, and longitudinal patient monitoring. It also examines the progress in VLFMs like Med-Flamingo, LLaVA-Med, and their zero-shot learning capabilities, and the exploration of parameter-efficient fine-tuning methods for efficient adaptation. This review concludes by encouraging the community to pursue these emergent and promising directions to strengthen the impact of artificial intelligence and deep learning on healthcare delivery and research.
学科与关键词
- Multimodal Large Language Models in Health Care:Applications,Challenges, and Future OutlookAlSaad, Rawan*; Abd-alrazaq, Alaa; Boughorbel, Sabri; Ahmed, Arfan; Renault, Max-Antoine; Damseh, Rafat; Sheikh, JavaidJOURNAL OF MEDICAL INTERNET RESEARCH, 2024, 26: e59505.被引 3
- Understanding natural language: Potential application of large language models to ophthalmologyYang, Zefeng; Wang, Deming; Zhou, Fengqi; Song, Diping; Zhang, Yinhang; Jiang, Jiaxuan; Kong, Kangjie; Liu, Xiaoyi; Qiao, Yu; Chang, Robert T.; Han, Ying; Li, Fei*; Tham, Clement C.*; Zhang, Xiulan*Asia - Pacific Journal of Ophthalmology, 2024, 13(4): 100085.
- A visual-language foundation model for computational pathologyLu, Ming Y.; Chen, Bowen; Williamson, Drew F. K.; Chen, Richard J.; Liang, Ivy; Ding, Tong; Jaume, Guillaume; Odintsov, Igor; Le, Long Phi; Gerber, Georg; Parwani, Anil V.; Zhang, Andrew; Mahmood, Faisal*Nature Medicine, 2024, 30(3).被引 39
