ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
会议论文

Scene Graph Generation using Depth-based Multimodal Network

Chen, Lianggangxu*; Lu, Jiale; Wang, Changbo; He, Gaoqi
IEEE International Conference on Multimedia and Expo (ICME)2023DOI: 10.1109/ICME55011.2023.00199
3阅读
引用收藏
分享

摘要

Scene graph generation (SGG) provides an efficient way for scene understanding. However, it has been plagued by the inaccurate classification of relative spatial relationship and incorrect feature information aggregation from distant objects. In this paper, we innovatively introduce the depth information of objects into SGG and propose a multimodal edge-featured graph attention network (MEGA-Net). MEGA-Net primarily comprises three modules. First, the edge-aware message passing (EMP) module extracts multimodal features and fuses them as edge features in the graph network via a quadrilinear model. Multimodal features consist of depth features, visual features, spatial features, and linguistic features. The depth feature in EMP provides the relative spatial relationship among objects which prevents the tail spatial predicates from being recognized as the head predicates. Second, we propose a depth-based self-supervised graph attention (DSGAT) module to predict the correlation probability between object pairs. By encoding the depth ranking of different object pairs in 2D images, DSGAT learns more accurate directional attention to avoid unrelated neighbors. Third, we introduce a predicate aware loss (PA-Loss) to alleviate the feature redundancy problem caused by extra depth information. This is achieved by introducing semantic frequency information that reflects the priority between different types of relationships. Systematic experiments show that our method achieves state-of-the-art performance on two popular datasets, VG and VRD.

学科与关键词

关键词
Scene Graph GenerationDepth InformationSelf-Supervised Graph Attention Network
相关
加载中…
  • GPS-Net: Graph Property Sensing Network for Scene Graph Generation
    Lin, Xin; Ding, Changxing; Zeng, Jinquan; Tao, DachengIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), ELECTR NETWORK, 2020-06-14 To 2020-06-19.被引 98
  • Knowledge-Enhanced Scene Graph Generation with Multimodal Relation Alignment (Student Abstract)
    Fu, Ze; Feng, Junhao; Zheng, Changmeng; Cai, Yi*36th AAAI Conference on Artificial Intelligence / 34th Conference on Innovative Applications of Artificial Intelligence / 12th Symposium on Educational Advances in Artificial Intelligence, ELECTR NETWORK, 2022-02-22 To 2022.
  • RU-Net: Regularized Unrolling Network for Scene Graph Generation
    Lin, Xin; Ding, Changxing*; Zhang, Jing; Zhan, Yibing; Tao, DachengIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, 2022-06-18 To 2022-06-24.
查看全部 →

会议信息

会议名称
IEEE International Conference on Multimedia and Expo (ICME)
会议日期
2023-07-10 至 2023-07-14
论文集名
2023 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO, ICME
发表日期
2023
起止页码
1139-1144
DOI
10.1109/ICME55011.2023.00199
资助标注
Natural Science Foundation of Chongqing [CSTB2022NSCQ-MSX0552]; National Natural Science Foundation of China [62002121, 62072183, 62102151]; Shanghai Science and Technology Commission [21511100700, 22511104600]; National Key Research and Development Program of China [2021ZD0111000]; Open Project Program of the State Key Lab of CAD&CG, Zhejiang University [A2203]
收录
ISTP

作者

  • C

    Chen, Lianggangxu (通讯) (第一作者)

  • L

    Lu, Jiale

查看全部 4 位作者收起作者
  • W

    Wang, Changbo

  • H

    He, Gaoqi

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417