摘要
Multimodal named entity recognition (MNER) is an emerging task that incorporates visual and textual inputs to detect named entities and predicts their corresponding entity types. However, existing MNER methods often fail to capture certain entity-related but text-loosely-related visual clues from the image, which may introduce task-irrelevant noises or even errors. To address this problem, we propose to utilize entity-related prompts for extracting proper visual clues with a pre-trained vision-language model. To better integrate different modalities and address the popular semantic gap problem, we further propose a modality-aware attention mechanism for better cross-modal fusion. Experimental results on two benchmarks show that our MNER approach outperforms the state-of-the-art MNER approaches with a large margin.
学科与关键词
- Multimodal Representation with Embedded Visual Guiding Objects for Named Entity Recognition in Social Media PostsWu, Zhiwei*; Zheng, Changmeng*; Cai, Yi; Chen, Junying; Leung, Ho-fung; Li, Qing28th ACM International Conference on Multimedia (MM), ELECTR NETWORK, 2020-10-12 To 2020-10-16.被引 19
- Object-Aware Multimodal Named Entity Recognition in Social Media Posts With Adversarial LearningZheng, Changmeng; Wu, Zhiwei; Wang, Tao; Cai, Yi*; Li, QingIEEE TRANSACTIONS ON MULTIMEDIA, 2021, 23: 2520-2532.被引 321
- Towards Bridged Vision and Language: Learning Cross-Modal Knowledge Representation for Relation ExtractionFeng, Junhao; Wang, Guohua; Zheng, Changmeng*; Cai, Yi; Fu, Ze; Wang, Yaowei; Wei, Xiao-Yong; Li, QingIEEE Transactions on Circuits and Systems for Video Technology, 2024, 34(1): 561-575.
