摘要
In the complex and multidimensional field of medicine, multimodal data are prevalent and crucial for informed clinical decisions.Multimodal data span a broad spectrum of data types, including medical images (eg, MRI and CT scans), time-series data (eg,sensor data from wearable devices and electronic health records), audio recordings (eg, heart and respiratory sounds and patientinterviews), text (eg, clinical notes and research articles), videos (eg, surgical procedures), and omics data (eg, genomics andproteomics). While advancements in large language models (LLMs) have enabled new applications for knowledge retrieval andprocessing in the medical field, most LLMs remain limited to processing unimodal data, typically text-based content, and oftenoverlook the importance of integrating the diverse data modalities encountered in clinical practice. This paper aims to present adetailed, practical, and solution-oriented perspective on the use of multimodal LLMs (M-LLMs) in the medical field. Ourinvestigation spanned M-LLM foundational principles, current and potential applications, technical and ethical challenges, andfuture research directions. By connecting these elements, we aimed to provide a comprehensive framework that links diverseaspects of M-LLMs, offering a unified vision for their future in health care. This approach aims to guide both future research andpractical implementations of M-LLMs in health care, positioning them as a paradigm shift toward integrated, multimodaldata-driven medical practice. We anticipate that this work will spark further discussion and inspire the development of innovativeapproaches in the next generation of medical M-LLM systems
学科与关键词
- Based on Medicine, The Now and Future of Large Language ModelsSu, Ziqing; Tang, Guozhang; Huang, Rui; Qiao, Yang; Zhang, Zheng; Dai, Xingliang*Cellular and Molecular Bioengineering, 2024, 17(4): 263-277.被引 1
- Visual-language foundation models in medicineLiu, Chunyu; Jin, Yixiao; Guan, Zhouyu; Li, Tingyao; Qin, Yiming; Qian, Bo; Jiang, Zehua; Wu, Yilan; Wang, Xiangning; Zheng, Ying Feng; Zeng, Dian*VISUAL COMPUTER, 2024.
- Artificial intelligence accelerates multi-modal biomedical process: A SurveyLi, Jiajia; Han, Xue; Qin, Yiming; Tan, Feng; Chen, Yulong; Wang, Zikai; Song, Haitao; Zhou, Xi; Zhang, Yuan; Hu, Lun; Hu, Pengwei*NEUROCOMPUTING, 2023, 558: 126720.被引 9
