摘要
Temporal action localization plays an important role in video analysis, which aims to localize and classify actions in untrimmed videos. Previous methods often predict actions on a feature space of a single temporal scale. However, the temporal features of a low-level scale lack sufficient semantics for action classification, while a high-level scale cannot provide the rich details of the action boundaries. In addition, the long-range dependencies of video frames are often ignored. To address these issues, a novel multitemporal-scale spatial-temporal transformer (MSST) network is proposed for temporal action localization, which predicts actions on a feature space of multiple temporal scales. Specifically, we first use refined feature pyramids of different scales to pass semantics from high-level scales to low-level scales. Second, to establish the long temporal scale of the entire video, we use a spatial-temporal transformer encoder to capture the long-range dependencies of video frames. Then, the refined features with long-range dependencies are fed into a classifier for coarse action prediction. Finally, to further improve the prediction accuracy, we propose a frame-level self-attention module to refine the classification and boundaries of each action instance. Most importantly, these three modules are jointly explored in a unified framework, and MSST has an anchor-free and end-to-end architecture. Extensive experiments show that the proposed method can outperform state-of-the-art approaches on the THUMOS14 dataset and achieve comparable performance on the ActivityNet1.3 dataset. Compared with A2Net (TIP20, Avg{0.3:0.7}), Sub-Action (CSVT2022, Avg{0.1:0.5}), and AFSD (CVPR21, Avg{0.3:0.7}) on the THUMOS14 dataset, the proposed method can achieve improvements of 12.6%, 17.4%, and 2.2%, respectively.
学科与关键词
- A Temporal-Aware Relation and Attention Network for Temporal Action LocalizationZhao, Yibo; Zhang, Hua; Gao, Zan*; Guan, Weili; Nie, Jie*; Liu, Anan; Wang, Meng; Chen, ShengyongIEEE Transactions on Image Processing, 2022, 31: 4746-4760.被引 16
- Relation Attention for Temporal Action LocalizationChen, Peihao; Gan, Chuang; Shen, Guangyao; Huang, Wenbing; Zeng, Runhao; Tan, Mingkui*IEEE TRANSACTIONS ON MULTIMEDIA, 2020, 22(10): 2723-2733.被引 40
- Graph Convolutional Module for Temporal Action Localization in VideosZeng, Runhao; Huang, Wenbing; Tan, Mingkui*; Rong, Yu; Zhao, Peilin; Huang, Junzhou; Gan, ChuangIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(10): 6209-6223.被引 20
