ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
期刊论文

A Multitemporal Scale and Spatial-Temporal Transformer Network for Temporal Action Localization

Gao, Zan; Cui, Xinglei; Zhuo, Tao*; Cheng, Zhiyong; Liu, An-An; Wang, Meng; Chen, Shenyong天津大学
IEEE Transactions on Human-Machine Systems2023-6DOI: 10.1109/THMS.2023.3266037
12阅读
引用收藏
分享

摘要

Temporal action localization plays an important role in video analysis, which aims to localize and classify actions in untrimmed videos. Previous methods often predict actions on a feature space of a single temporal scale. However, the temporal features of a low-level scale lack sufficient semantics for action classification, while a high-level scale cannot provide the rich details of the action boundaries. In addition, the long-range dependencies of video frames are often ignored. To address these issues, a novel multitemporal-scale spatial-temporal transformer (MSST) network is proposed for temporal action localization, which predicts actions on a feature space of multiple temporal scales. Specifically, we first use refined feature pyramids of different scales to pass semantics from high-level scales to low-level scales. Second, to establish the long temporal scale of the entire video, we use a spatial-temporal transformer encoder to capture the long-range dependencies of video frames. Then, the refined features with long-range dependencies are fed into a classifier for coarse action prediction. Finally, to further improve the prediction accuracy, we propose a frame-level self-attention module to refine the classification and boundaries of each action instance. Most importantly, these three modules are jointly explored in a unified framework, and MSST has an anchor-free and end-to-end architecture. Extensive experiments show that the proposed method can outperform state-of-the-art approaches on the THUMOS14 dataset and achieve comparable performance on the ActivityNet1.3 dataset. Compared with A2Net (TIP20, Avg{0.3:0.7}), Sub-Action (CSVT2022, Avg{0.1:0.5}), and AFSD (CVPR21, Avg{0.3:0.7}) on the THUMOS14 dataset, the proposed method can achieve improvements of 12.6%, 17.4%, and 2.2%, respectively.

学科与关键词

关键词
TransformersSemanticsFeature extractionProposalsLocation awarenessConvolutionTask analysisFrame-level self-attention (FSA)multiple temporal scalesrefined feature pyramids (RFPs)spatial-temporal transformer (STT)temporal action localization (TAL)
相关
加载中…
  • A Temporal-Aware Relation and Attention Network for Temporal Action Localization
    Zhao, Yibo; Zhang, Hua; Gao, Zan*; Guan, Weili; Nie, Jie*; Liu, Anan; Wang, Meng; Chen, ShengyongIEEE Transactions on Image Processing, 2022, 31: 4746-4760.被引 16
  • Relation Attention for Temporal Action Localization
    Chen, Peihao; Gan, Chuang; Shen, Guangyao; Huang, Wenbing; Zeng, Runhao; Tan, Mingkui*IEEE TRANSACTIONS ON MULTIMEDIA, 2020, 22(10): 2723-2733.被引 40
  • Graph Convolutional Module for Temporal Action Localization in Videos
    Zeng, Runhao; Huang, Wenbing; Tan, Mingkui*; Rong, Yu; Zhao, Peilin; Huang, Junzhou; Gan, ChuangIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(10): 6209-6223.被引 20
查看全部 →

发表信息

论文状态
公开发表
期刊
IEEE Transactions on Human-Machine Systems
发表日期
2023-06
卷期
第 53 卷, 第 3 期
页码
569-580
DOI
10.1109/THMS.2023.3266037
语言
ENGLISH
成果收录
SCIE
资助标注
National Natural Science Foundation of China [61872270, 62171145, 61906108, 62020106004, 92048301]; Young Creative Team in Universities of Shandong Province [2020KJN012]; Jinan 20 Projects in Universities [2020GXRC040]; Shandong Project Towards the Integration of Education and Industry [2022PYI001, 2022PY009, 2022JBZ01-03]; Tianjin Education Committee Science and Technology Development Foundation [2017KJ254]

作者

  • G

    Gao, Zan (第一作者)

    tianjin univ technol · shandong acad sci

  • C

    Cui, Xinglei

    shandong acad sci

查看全部 7 位作者收起作者
  • Z

    Zhuo, Tao (通讯)

    shandong acad sci

  • C

    Cheng, Zhiyong

    shandong acad sci

  • L

    Liu, An-An

    tianjin univ

  • W

    Wang, Meng

  • C

    Chen, Shenyong

    tianjin univ technol

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417