摘要
Existing scene graph generation methods suffer the limitations when the image lacks of sufficient visual contexts. To address this limitation, we propose a knowledge-enhanced scene graph generation model with multimodal relation alignment, which supplements the missing visual contexts by well-aligned textual knowledge. First, we represent the textual information into contextualized knowledge which is guided by the visual objects to enhance the contexts. Furthermore, we align the multimodal relation triplets by co-attention module for better semantics fusion. The experimental results show the effectiveness of our method.
- DRAKE: Deep Pair-Wise Relation Alignment for Knowledge-Enhanced Multimodal Scene Graph Generation in Social Media PostsFu, Ze; Zheng, Changmeng; Feng, Junhao; Cai, Yi*; Wei, Xiao-Yong; Wang, Yaowei; Li, QingIEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(7): 3199-3213.
- Scene Graph Generation using Depth-based Multimodal NetworkChen, Lianggangxu*; Lu, Jiale; Wang, Changbo; He, GaoqiIEEE International Conference on Multimedia and Expo (ICME), Brisbane, AUSTRALIA, 2023-07-10 To 2023-07-14.
- Improving rare relation inferring for scene graph generation using bipartite graph networkLu, Jiale; Chen, Lianggangxu; Guan, Haoyue; Lin, Shaohui; Gu, Chunhua; Wang, Changbo; He, Gaoqi*Computer Vision and Image Understanding, 2024, 239: 103901.
