摘要
Video question answering is an important task combining both Natural Language Processing and Computer Vision, which requires a machine to obtain a thorough understanding of the video. Most existing approaches simply capture spatio-temporal information in videos by using a combination of recurrent and convolutional neural networks. Nonetheless, most previous work focus on only salient frames or regions, which normally lacks some significant details, such as potential location and action relations. In this paper, we propose a new method called Graph-based Multi-interaction Network for video question answering. In our model, a new attention mechanism named multi-interaction is designed to capture both element-wise and segment-wise sequence interactions simultaneously, which can be found between and inside the multi-modal inputs. Moreover, we propose a graph-based relation-aware neural network to explore a more fine-grained visual representation, which could explore the relationships and dependencies between objects spatially and temporally. We evaluate our method on TGIF-QA and other two video QA datasets. The qualitative and quantitative experimental results show the effectiveness of our model, which achieves state-of-the-art performance.
学科与关键词
- Location-Aware Graph Convolutional Networks for Video Question AnsweringHuang, Deng; Chen, Peihao; Zeng, Runhao; Du, Qing; Tan, Mingkui*; Gan, Chuang34th AAAI Conference on Artificial Intelligence / 32nd Innovative Applications of Artificial Intelligence Conference / 10th AAAI Symposium on Educational Advances in Artificial Intelligence, New York, NY, 2020-02-07 To 2020-02-12.被引 63
- Differentiated Attention with Multi-modal Reasoning for Video Question AnsweringYao, Shentao; Li, Kun*; Xing, Kun; Wu, Kewei; Xie, Zhao; Guo, DanIEEE International Conference on Electrical Engineering, Big Data and Algorithms (EEBDA), Changchun, PEOPLES R CHINA, 2022-02-25 To 2022-02-27.
- A question-guided multi-hop reasoning graph network for visual question answeringXu, Zhaoyang; Gu, Jinguang; Liu, Maofu; Zhou, Guangyou; Fu, Haidong; Qiu, Chen*INFORMATION PROCESSING & MANAGEMENT, 2023, 60(2): 103207.被引 3
