摘要
With the rapid development of social networks, video data has been growing explosively. As one of the important social mediums, spatiotemporal characteristics of videos have attracted considerable attention in recommendation system and video understanding. In this paper, we discuss the video moment retrieval (VMR) task, which locates moments in a video based on different textual queries. Existing methods are of two pipelines: 1) proposal-free approaches are mainly in modifying multi-modal interaction strategy; 2) proposal-based methods are dedicated to designing advanced proposal generation paradigm. Recently, contrastive representation learning has been successfully applied to the field of video understanding. From a new perspective, we propose a new VMR framework, named spatiotemporal contrastive network (STCNet), to learn discriminative boundary features of video grounding by contrast learning. To be specific, we propose a boundary matching sampling module for dense negative sample sampling. The contrast learning can refine the feature representations in the training phase without any additional cost in inference. On three public datasets, Charades-STA, ActivityNet Captions and TACoS, our proposed method performs competitive performance.
学科与关键词
- Video Moment Retrieval With Cross-Modal Neural Architecture SearchYang, Xun; Wang, Shanshan; Dong, Jian; Dong, Jianfeng; Wang, Meng; Chua, Tat-SengIEEE Transactions on Image Processing, 2022, 31: 1204-1216.被引 23
- A Survey on Video Moment LocalizationLiu, Meng; Nie, Liqiang*; Wang, Yunxiao; Wang, Meng; Rui, YongACM Computing Surveys, 2023, 55(9): 188.
- Contrastive Spatio-Temporal Pretext Learning for Self-Supervised Video RepresentationZhang, Yujia*; Po, Lai-Man; Xu, Xuyuan; Liu, Mengyang; Wang, Yexin; Ou, Weifeng; Zhao, Yuzhi; Yu, Wing-Yin36th AAAI Conference on Artificial Intelligence / 34th Conference on Innovative Applications of Artificial Intelligence / 12th Symposium on Educational Advances in Artificial Intelligence, ELECTR NETWORK, 2022-02-22 To 2022.被引 3
