摘要
Computational pathology1,2 has witnessed considerable progress in the development of both task-specific predictive models and task-agnostic self-supervised vision encoders3,4. However, despite the explosive growth of generative artificial intelligence (AI), there have been few studies on building general-purpose multimodal AI assistants and copilots5 tailored to pathology. Here we present PathChat, a vision-language generalist AI assistant for human pathology. We built PathChat by adapting a foundational vision encoder for pathology, combining it with a pretrained large language model and fine-tuning the whole system on over 456,000 diverse visual-language instructions consisting of 999,202 question and answer turns. We compare PathChat with several multimodal vision-language AI assistants and GPT-4V, which powers the commercially available multimodal general-purpose AI assistant ChatGPT-4 (ref. 6). PathChat achieved state-of-the-art performance on multiple-choice diagnostic questions from cases with diverse tissue origins and disease models. Furthermore, using open-ended questions and human expert evaluation, we found that overall PathChat produced more accurate and pathologist-preferable responses to diverse queries related to pathology. As an interactive vision-language AI copilot that can flexibly handle both visual and natural language inputs, PathChat may potentially find impactful applications in pathology education, research and human-in-the-loop clinical decision-making. PathChat, a multimodal generative AI copilot for human pathology, has been trained on a large dataset of visual-language instructions to interactively assist users with diverse pathology tasks.
学科与关键词
- A visual-language foundation model for computational pathologyLu, Ming Y.; Chen, Bowen; Williamson, Drew F. K.; Chen, Richard J.; Liang, Ivy; Ding, Tong; Jaume, Guillaume; Odintsov, Igor; Le, Long Phi; Gerber, Georg; Parwani, Anil V.; Zhang, Andrew; Mahmood, Faisal*Nature Medicine, 2024, 30(3).被引 39
- A vision-language foundation model for precision oncologyXiang, Jinxi; Wang, Xiyue; Zhang, Xiaoming; Xi, Yinghua; Eweje, Feyisope; Chen, Yijiang; Li, Yuchen; Bergstrom, Colin; Gopaulchan, Matthew; Kim, Ted; Yu, Kun-Hsing; Willens, Sierra; Olguin, Francesca Maria; Nirschl, Jeffrey J.; Neal, Joel; Diehn, Maximilian; Yang, Sen*; Li, Ruijiang*Nature, 2025.
- Towards a general-purpose foundation model for computational pathologyChen, Richard J.; Ding, Tong; Lu, Ming Y.; Williamson, Drew F. K.; Jaume, Guillaume; Song, Andrew H.; Chen, Bowen; Zhang, Andrew; Shao, Daniel; Shaban, Muhammad; Williams, Mane; Oldenburg, Lukas; Weishaupt, Luca L.; Wang, Judy J.; Vaidya, Anurag; Le, Long Phi; Gerber, Georg; Sahai, Sharifa; Williams, Walt; Mahmood, Faisal*Nature Medicine, 2024, 30(3): 850-862.被引 84
