摘要
Visual contexts often help to recognize named entities more precisely in short texts such as tweets or snapchat. For example, one can identify "Charlie" as a name of a dog according to the user posts. Previous works on multimodal named entity recognition ignore the corresponding relations of visual objects and entities. Visual objects are considered as fine-grained image representations. For a sentence with multiple entity types, objects of the relevant image can be utilized to capture different entity information. In this paper, we propose a neural network which combines object-level image information and character-level text information to predict entities. Vision and language are bridged by leveraging object labels as embeddings, and a dense co-attention mechanism is introduced for fine-grained interactions. Experimental results in Twitter dataset demonstrate that our method outperforms the state-of-the-art methods.
学科与关键词
- Object-Aware Multimodal Named Entity Recognition in Social Media Posts With Adversarial LearningZheng, Changmeng; Wu, Zhiwei; Wang, Tao; Cai, Yi*; Li, QingIEEE TRANSACTIONS ON MULTIMEDIA, 2021, 23: 2520-2532.被引 321
- PromptMNER: Prompt-Based Entity-Related Visual Clue Extraction and Integration for Multimodal Named Entity RecognitionWang, Xuwu; Tian, Junfeng; Gui, Min; Li, Zhixu*; Ye, Jiabo; Yan, Ming; Xiao, Yanghua*27th International Conference on Database Systems for Advanced Applications (DASFAA), IIIT Hyderabad, ELECTR NETWORK, 2022-04-11 To 2022-04-14.被引 6
- Towards Bridged Vision and Language: Learning Cross-Modal Knowledge Representation for Relation ExtractionFeng, Junhao; Wang, Guohua; Zheng, Changmeng*; Cai, Yi; Fu, Ze; Wang, Yaowei; Wei, Xiao-Yong; Li, QingIEEE Transactions on Circuits and Systems for Video Technology, 2024, 34(1): 561-575.
