ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
期刊论文

Dual Encoding for Video Retrieval by Text

Dong, Jianfeng; Li, Xirong*; Xu, Chaoxi; Yang, Xun; Yang, Gang; Wang, Xun*; Wang, Mengzhejiang gongshang univ; renmin univ china
IEEE Transactions on Pattern Analysis and Machine Intelligence2022-8DOI: 10.1109/TPAMI.2021.3059295
65被引12阅读
引用收藏
分享

摘要

This paper attacks the challenging problem of video retrieval by text. In such a retrieval paradigm, an end user searches for unlabeled videos by ad-hoc queries described exclusively in the form of a natural-language sentence, with no visual example provided. Given videos as sequences of frames and queries as sequences of words, an effective sequence-to-sequence cross-modal matching is crucial. To that end, the two modalities need to be first encoded into real-valued vectors and then projected into a common space. In this paper we achieve this by proposing a dual deep encoding network that encodes videos and queries into powerful dense representations of their own. Our novelty is two-fold. First, different from prior art that resorts to a specific single-level encoder, the proposed network performs multi-level encoding that represents the rich content of both modalities in a coarse-to-fine fashion. Second, different from a conventional common space learning algorithm which is either concept based or latent space based, we introduce hybrid space learning which combines the high performance of the latent space and the good interpretability of the concept space. Dual encoding is conceptually simple, practically effective and end-to-end trained with hybrid space learning. Extensive experiments on four challenging video datasets show the viability of the new method. Code and data are available at https://github.com/danieljf24/hybrid_space.

学科与关键词

关键词
EncodingVisualizationFeature extractionComputational modelingLinguisticsElectronic mailRecurrent neural networksVideo retrievalcross-modal representation learningdual encodinghybrid space learning
相关
加载中…

文献加载失败,请重试

重试

发表信息

论文状态
公开发表
期刊
IEEE Transactions on Pattern Analysis and Machine Intelligence
发表日期
2022-08
卷期
第 44 卷, 第 8 期
页码
4065-4080
DOI
10.1109/TPAMI.2021.3059295
语言
ENGLISH
期刊等级
1区
成果收录
SCIE
资助标注
NSFC [61902347, 61672523]; BJNSF [4202033]; ZJNSF [LQ19F020002]; Public Welfare Technology Research Project of Zhejiang Province [LGF21F020010]; Fundamental Research Funds for the Central Universities; Research Funds of Renmin University of China [18XNLG19]; Research Program of Zhejiang Lab [2019KD0AC02]; Public Computing Cloud of Renmin University of China; Alibaba-ZJU Joint Research Institute of Frontier Technologies

作者

  • D

    Dong, Jianfeng (第一作者)

    zhejiang gongshang univ

  • L

    Li, Xirong (通讯)

    renmin univ china

查看全部 7 位作者收起作者
  • X

    Xu, Chaoxi

    renmin univ china

  • Y

    Yang, Xun

  • Y

    Yang, Gang

    renmin univ china

  • W

    Wang, Xun (通讯)

    zhejiang gongshang univ

  • W

    Wang, Meng

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417