摘要
Spatiotemporal attention learning for video question answering (VideoQA) has always been a challenging task, where existing approaches treat the attention parts and the nonattention parts in isolation. In this work, we propose to enforce the correlation between the attention parts and the nonattention parts as a distance constraint for discriminative spatiotemporal attention learning. Specifically, we first introduce a novel attention-guided erasing mechanism in the traditional spatiotemporal attention to obtain multiple aggregated attention features and nonattention features and then learn to separate the attention and the nonattention features with an appropriate distance. The distance constraint is enforced by a metric learning loss, without increasing the inference complexity. In this way, the model can learn to produce more discriminative spatiotemporal attention distribution on videos, thus enabling more accurate question answering. In order to incorporate the multiscale spatiotemporal information that is beneficial for video understanding, we additionally develop a pyramid variant on basis of the proposed approach. Comprehensive ablation experiments are conducted to validate the effectiveness of our approach, and state-of-the-art performance is achieved on several widely used datasets for VideoQA.
学科与关键词
- Graph-Based Multi-Interaction Network for Video Question AnsweringGu, Mao; Zhao, Zhou*; Jin, Weike; Hong, Richang; Wu, FeiIEEE Transactions on Image Processing, 2021, 30: 2758-2770.被引 29
- Differentiated Attention with Multi-modal Reasoning for Video Question AnsweringYao, Shentao; Li, Kun*; Xing, Kun; Wu, Kewei; Xie, Zhao; Guo, DanIEEE International Conference on Electrical Engineering, Big Data and Algorithms (EEBDA), Changchun, PEOPLES R CHINA, 2022-02-25 To 2022-02-27.
- Location-Aware Graph Convolutional Networks for Video Question AnsweringHuang, Deng; Chen, Peihao; Zeng, Runhao; Du, Qing; Tan, Mingkui*; Gan, Chuang34th AAAI Conference on Artificial Intelligence / 32nd Innovative Applications of Artificial Intelligence Conference / 10th AAAI Symposium on Educational Advances in Artificial Intelligence, New York, NY, 2020-02-07 To 2020-02-12.被引 63
