ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
期刊论文

Stacked Multimodal Attention Network for Context-Aware Video Captioning

Zheng, Yi; Zhang, Yuejie*; Feng, Rui; Zhang, Tao*; Fan, Weiguo复旦大学
IEEE Transactions on Circuits and Systems for Video Technology2022-1DOI: 10.1109/TCSVT.2021.3058626
12被引5阅读
引用收藏
分享

摘要

Recent neural models for video captioning usually employ an attention-based encoder-decoder framework. However, current approaches mainly attend to the motion features and object features of the video when generating the caption, but ignore the potential but useful historical information. Besides, exposure bias and vanishing gradients problems always exist in current caption generation models. In this paper, we propose a novel video captioning framework, named Stacked Multimodal Attention Network (SMAN). It adopts additional visual and textual historical information during caption generation as context features, employs a stacked architecture to process different features gradually, and utilizes the Reinforcement Learning method and coarse-to-fine training strategy to further improve the generated results. Both quantitative and qualitative experiments on the benchmark datasets of MSVD and MSR-VTT show the effectiveness and feasibility of our framework. The codes are available on https://github.com/zhengyi123456/SMAN.

学科与关键词

关键词
Feature extractionVisualizationDecodingTrainingBiological system modelingContext modelingPredictive modelsVideo captioningcontext-awarestacked multimodal attention networkcoarse-to-fine trainingreinforcement learning
相关
加载中…
  • Describing Video With Attention-Based Bidirectional LSTM
    Bin, Yi; Yang, Yang*; Shen, Fumin; Xie, Ning; Shen, Heng Tao; Li, XuelongIEEE Transactions on Cybernetics, 2019, 49(7): 2631-2641.被引 11
  • Video captioning with text -based dynamic attention and step-by-step learning
    Xiao, Huanhou; Shi, Jinglun*PATTERN RECOGNITION LETTERS, 2020, 133: 305-312.被引 10
  • Exploring diverse and fine-grained caption for video by incorporating convolutional architecture into LSTM-based model
    Xiao, Huanhou; Xu, Junwei; Shi, Jinglun*PATTERN RECOGNITION LETTERS, 2020, 129: 173-180.被引 9
查看全部 →

发表信息

论文状态
公开发表
期刊
IEEE Transactions on Circuits and Systems for Video Technology
发表日期
2022-01
卷期
第 32 卷, 第 1 期
页码
31-42
DOI
10.1109/TCSVT.2021.3058626
语言
ENGLISH
期刊等级
1区
成果收录
SCIE
资助标注
National Natural Science Foundation of China [61976057, 61572140]; Science and Technology Development Plan of Shanghai Science and Technology Commission [20511101203, 20511102702, 20511101403, 19DZ2205700, 2021SHZDZX0103]; Shanghai Natural Science Foundation [19ZR1417200]; Humanities and Social Sciences Planning Fund of Ministry of Education of China [19YJA630116]

作者

  • Z

    Zheng, Yi (第一作者)

    fudan univ

  • Z

    Zhang, Yuejie (通讯)

    fudan univ

查看全部 5 位作者收起作者
  • F

    Feng, Rui

    fudan univ

  • Z

    Zhang, Tao (通讯)

  • F

    Fan, Weiguo

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417