ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
期刊论文

DRAKE: Deep Pair-Wise Relation Alignment for Knowledge-Enhanced Multimodal Scene Graph Generation in Social Media Posts

Fu, Ze; Zheng, Changmeng; Feng, Junhao; Cai, Yi*; Wei, Xiao-Yong; Wang, Yaowei; Li, Qing四川大学; y
IEEE Transactions on Circuits and Systems for Video Technology2023-7DOI: 10.1109/TCSVT.2022.3231437
6阅读
引用收藏
分享

摘要

Scene Graph Generation (SGG) is a typical computer vision task that detects objects and corresponding predicates in an image. Existing SGG methods focus on modeling visual contexts to generate scene graphs and are conducted on well-annotated datasets with high-quality images. However, the quality is unguaranteed for images in social media posts, so that some images may be incomplete or occluded by some obstacles, hence might not provide sufficient visual context for SGG. Therefore, previous methods might result in missing or false visual relationship detection due to lacking visual contexts. To effectively generate the scene graphs in social media, we study multimodal scene graph generation (MSG) in this paper. MSG aims to develop visual scene graphs from images in social media posts with the support of text sentences. However, leveraging textual contents by simple multimodal alignment such as object-level alignment neglects the inherent pair-wise mapping between multimodal object pairs. To address the limitations, we propose a method named Deep pair-wise Relation Alignment for Knowledge-Enhanced (DRAKE) multimodal scene graph generation. The model supplements the missing visual contexts with well-aligned textual knowledge. It first represents the textual information into object-aware knowledge representation with the help of vision data. Furthermore, our proposed DRAKE facilitates the interaction of the info between multimodal pair-wise representations. A multimodal context enhancement layer can be devised to help the model generate the scene graph. To evaluate the model performance of SGG on social media images, we propose a social media SGG dataset called MSG. We comprehensively analyze the effectiveness of our proposed method on the MSG dataset. The experimental results on the MSG dataset indicate that our model outperforms the previous methods. To fairly compare our method with other SGG models, we also conduct experiments on the Visual Genome dataset for more analysis The MSG dataset is released on https://github.com/FuZe4ever/MSG.

学科与关键词

关键词
Scene graph generationsocial media postsknowledge enhancementpair-wise alignment
相关
加载中…
  • Knowledge-Enhanced Scene Graph Generation with Multimodal Relation Alignment (Student Abstract)
    Fu, Ze; Feng, Junhao; Zheng, Changmeng; Cai, Yi*36th AAAI Conference on Artificial Intelligence / 34th Conference on Innovative Applications of Artificial Intelligence / 12th Symposium on Educational Advances in Artificial Intelligence, ELECTR NETWORK, 2022-02-22 To 2022.
  • Scene Graph Generation using Depth-based Multimodal Network
    Chen, Lianggangxu*; Lu, Jiale; Wang, Changbo; He, GaoqiIEEE International Conference on Multimedia and Expo (ICME), Brisbane, AUSTRALIA, 2023-07-10 To 2023-07-14.
  • Towards Bridged Vision and Language: Learning Cross-Modal Knowledge Representation for Relation Extraction
    Feng, Junhao; Wang, Guohua; Zheng, Changmeng*; Cai, Yi; Fu, Ze; Wang, Yaowei; Wei, Xiao-Yong; Li, QingIEEE Transactions on Circuits and Systems for Video Technology, 2024, 34(1): 561-575.
查看全部 →

发表信息

论文状态
公开发表
期刊
IEEE Transactions on Circuits and Systems for Video Technology
发表日期
2023-07
卷期
第 33 卷, 第 7 期
页码
3199-3213
DOI
10.1109/TCSVT.2022.3231437
语言
ENGLISH
成果收录
SCIE
资助标注
Fundamental Research Funds for the Central Universities, South China University of Technology (SCUT) [D2200150, D2201300, D2210010]; National Natural Science Foundation of China [62076100, 61872256]; Science and Technology Planning Project of Guangdong Province [2020B0101100002]; Hong Kong Research Grants Council through a General Research Fund [11204919]; Peng Cheng Laboratory Key Research Project [PCL2021407]

作者

  • F

    Fu, Ze (第一作者)

  • Z

    Zheng, Changmeng

    hong kong polytech univ

查看全部 7 位作者收起作者
  • F

    Feng, Junhao

  • C

    Cai, Yi (通讯)

  • W

    Wei, Xiao-Yong

    sichuan univ

  • W

    Wang, Yaowei

  • L

    Li, Qing

    hong kong polytech univ

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417