ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
期刊论文

Video-based spatio-temporal scene graph generation with efficient self-supervision tasks

Chen, Lianggangxu; Cai, Yiqing; Lu, Changhong; Wang, Changbo; He, Gaoqi*
MULTIMEDIA TOOLS AND APPLICATIONS2023-10DOI: 10.1007/s11042-023-14640-6
12阅读
引用收藏
分享

摘要

Spatio-temporal Scene Graphs Generation (STSGG) aims to extract a sequence of graph-based semantic representations for high-level visual tasks. Existing works often fail to exploit the strong temporal correlation and the details of local features, which leads to the inability to distinguish the action between dynamic relation (e.g., drinking) and static relation (e.g., holding). Furthermore, due to bad long-tailed bias, the prediction results are troubled by inaccurate tail predicates classifications. To address these issues, a slowfast local-aware attention (SFLA) Network is proposed for temporal modeling in STSGG. First, a two-branch network is used to extract static and dynamic relation features respectively. Second, a local relation-aware attention (LRA) module is proposed to attach higher importance to the crucial elements in the local relationship. Third, three novel self-supervision prediction tasks are proposed, that is, spatial location, human attention state, and distance variation. Such self-supervision tasks are trained simultaneously with the main model to alleviate the long-tailed bias problem and enhance feature discrimination. Systematic experiments show that our method achieves state-of-the-art performance in the recently proposed Action Genome (AG) dataset and the popular ImageNet Video dataset.

学科与关键词

关键词
Spatio-temporal scene graphs generationSelf-supervisionLocal relation-aware attention
相关
加载中…
  • Scene Graph Generation using Depth-based Multimodal Network
    Chen, Lianggangxu*; Lu, Jiale; Wang, Changbo; He, GaoqiIEEE International Conference on Multimedia and Expo (ICME), Brisbane, AUSTRALIA, 2023-07-10 To 2023-07-14.
  • Relation Attention for Temporal Action Localization
    Chen, Peihao; Gan, Chuang; Shen, Guangyao; Huang, Wenbing; Zeng, Runhao; Tan, Mingkui*IEEE TRANSACTIONS ON MULTIMEDIA, 2020, 22(10): 2723-2733.被引 40
  • Graph-Based Multi-Interaction Network for Video Question Answering
    Gu, Mao; Zhao, Zhou*; Jin, Weike; Hong, Richang; Wu, FeiIEEE Transactions on Image Processing, 2021, 30: 2758-2770.被引 29
查看全部 →

发表信息

论文状态
公开发表
期刊
MULTIMEDIA TOOLS AND APPLICATIONS
发表日期
2023-10
卷期
第 82 卷, 第 25 期
页码
38947-38966
DOI
10.1007/s11042-023-14640-6
语言
ENGLISH
成果收录
SCIE
资助标注
Natural Science Foundation of Chongqing [CSTB2022NSCQ-MSX0552]; National Natural Science Foundation of China [62002121, 62072183, 62102151]; Shanghai Science and Technology Commission [21511100700, 22511104600]; National Key Research and Development Program of China [2021ZD0111000]; Research Project of Shanghai Science and Technology Commission [20DZ2260300]; Shanghai Sailing Program [21YF1411200]; CAAI-Huawei MindSpore Open Fund [CAAIXSJLJJ-2021-031A]; Open Project Program of the State Key Lab of CADCG [A2203]

作者

  • C

    Chen, Lianggangxu (第一作者)

  • C

    Cai, Yiqing

查看全部 5 位作者收起作者
  • L

    Lu, Changhong

  • W

    Wang, Changbo

  • H

    He, Gaoqi (通讯)

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417