摘要
In natural language processing, relation extraction (RE) is to detect and classify the semantic relationship of two given entities within a sentence. Previous RE methods consider only the textual contents and suffer performance decline in social media when texts lack contexts. Incorporating text-related visual information can supplement the missing semantics for relation extraction in social media posts. However, textual relations are usually abstract and of high-level semantics, which causes the semantic gap between visual contents and textual expressions. In this paper, we propose RECK - a neural network for relation extraction with cross-modal knowledge representations. Different from previous multimodal methods training a common subspace for all modalities, we bridge the semantic gaps by explicitly selecting knowledge paths from external knowledge through the cross-modal object-entity pairs. We further extend the paths into a knowledge graph, and adopt a graph attention network to capture the multi-grained relevant concepts which can provide higher level and key semantics information from external knowledge. Besides, we employ a cross-modal attention mechanism to align and fuse the multimodal information. Experimental results on a multimodal RE dataset show that our model achieves new state-of-the-art performance with knowledge evidence.
学科与关键词
- Multimodal Representation with Embedded Visual Guiding Objects for Named Entity Recognition in Social Media PostsWu, Zhiwei*; Zheng, Changmeng*; Cai, Yi; Chen, Junying; Leung, Ho-fung; Li, Qing28th ACM International Conference on Multimedia (MM), ELECTR NETWORK, 2020-10-12 To 2020-10-16.被引 19
- Object-Aware Multimodal Named Entity Recognition in Social Media Posts With Adversarial LearningZheng, Changmeng; Wu, Zhiwei; Wang, Tao; Cai, Yi*; Li, QingIEEE TRANSACTIONS ON MULTIMEDIA, 2021, 23: 2520-2532.被引 321
- DRAKE: Deep Pair-Wise Relation Alignment for Knowledge-Enhanced Multimodal Scene Graph Generation in Social Media PostsFu, Ze; Zheng, Changmeng; Feng, Junhao; Cai, Yi*; Wei, Xiao-Yong; Wang, Yaowei; Li, QingIEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(7): 3199-3213.
