摘要
The accelerated adoption of digital pathology and advances in deep learning have enabled the development of robust models for various pathology tasks across a diverse array of diseases and patient cohorts. However, model training is often difficult due to label scarcity in the medical domain, and a model's usage is limited by the specific task and disease for which it is trained. Additionally, most models in histopathology leverage only image data, a stark contrast to how humans teach each other and reason about histopathologic entities. We introduce CONtrastive learning from Captions for Histopathology (CONCH), a visual-language foundation model developed using diverse sources of histopathology images, biomedical text and, notably, over 1.17 million image-caption pairs through task-agnostic pretraining. Evaluated on a suite of 14 diverse benchmarks, CONCH can be transferred to a wide range of downstream tasks involving histopathology images and/or text, achieving state-of-the-art performance on histology image classification, segmentation, captioning, and text-to-image and image-to-text retrieval. CONCH represents a substantial leap over concurrent visual-language pretrained systems for histopathology, with the potential to directly facilitate a wide array of machine learning-based workflows requiring minimal or no further supervised fine-tuning. Developed using diverse sources of histopathology images, biomedical text and over 1.17 million image-caption pairs, evaluated on a suite of 14 diverse benchmarks, a visual-language foundation model achieves state-of-the-art performance on a wide array of clinically relevant pathology tasks.
学科与关键词
- Towards a general-purpose foundation model for computational pathologyChen, Richard J.; Ding, Tong; Lu, Ming Y.; Williamson, Drew F. K.; Jaume, Guillaume; Song, Andrew H.; Chen, Bowen; Zhang, Andrew; Shao, Daniel; Shaban, Muhammad; Williams, Mane; Oldenburg, Lukas; Weishaupt, Luca L.; Wang, Judy J.; Vaidya, Anurag; Le, Long Phi; Gerber, Georg; Sahai, Sharifa; Williams, Walt; Mahmood, Faisal*Nature Medicine, 2024, 30(3): 850-862.被引 84
- A pathology foundation model for cancer diagnosis and prognosis predictionWang, Xiyue; Zhao, Junhan; Marostica, Eliana; Yuan, Wei; Jin, Jietian; Zhang, Jiayu; Li, Ruijiang; Tang, Hongping; Wang, Kanran; Li, Yu; Wang, Fang; Peng, Yulong; Zhu, Junyou; Zhang, Jing; Jackson, Christopher R.; Zhang, Jun; Dillon, Deborah; Lin, Nancy U.; Sholl, Lynette; Denize, Thomas; Meredith, David; Ligon, Keith L.; Signoretti, Sabina; Ogino, Shuji; Golden, Jeffrey A.; Nasrallah, MacLean P.; Han, Xiao; Yang, Sen*; Yu, Kun-Hsing*Nature, 2024.被引 9
- A multimodal generative AI copilot for human pathologyLu, Ming Y.; Chen, Bowen; Williamson, Drew F. K.; Chen, Richard J.; Zhao, Melissa; Chow, Aaron K.; Ikemura, Kenji; Kim, Ahrong; Pouli, Dimitra; Patel, Ankush; Soliman, Amr; Chen, Chengkuan; Ding, Tong; Wang, Judy J.; Gerber, Georg; Liang, Ivy; Le, Long Phi; Parwani, Anil V.; Weishaupt, Luca L.; Mahmood, Faisal*Nature, 2024, 634(8033).被引 26
