ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
会议论文

Differentiated Attention with Multi-modal Reasoning for Video Question Answering

Yao, Shentao; Li, Kun*; Xing, Kun; Wu, Kewei; Xie, Zhao; Guo, Dan
IEEE International Conference on Electrical Engineering, Big Data and Algorithms (EEBDA)2022DOI: 10.1109/EEBDA53927.2022.9744732
9阅读
引用收藏
分享

摘要

It is extremely challenging to infer the answers for long questions and complex videos. Video question answering not only needs to capture clues from questions but also reasonably infer the certain clip or frame in videos. In the paper, we propose a novel method to understand the entire video and reasonably infer the answer to the question. We utilize a traditional attention mechanism combined with the multi-head structure to construct a differentiated attention module. Different from existing methods, our method is dedicated to obtaining differentiated features. Videos on our method are split into a few video clips, and there is a great overlap between video clips. Thus, simple using the selfattention mechanism to aggregate features will lead to excessive redundancy in the captured features. To tackle this issue, we propose a differentiated attention module consists of traditional attention mechanism and multi-head structure to focus on the core semantics and decode different clips or phrases. In addition, we also apply the differentiated attention block on question aggregation and video clues reasoning. We use different query attention loss (DQALoss) to solve the problem of question requiring stronger differentiation. Meanwhile, we propose to utilize the multi-modal factorized bilinear pooling method to solve multi-modal features reasoning and interaction. Our experiment shows that the proposed method outperforms existing methods on TGIF-QA datasets by large margins. The experimental results show the effectiveness of our method.

学科与关键词

关键词
video question answeringdifferentiated attentionmulti-modal fusionmulti-modal interaction
相关
加载中…

文献加载失败,请重试

重试

会议信息

会议名称
IEEE International Conference on Electrical Engineering, Big Data and Algorithms (EEBDA)
会议日期
2022-02-25 至 2022-02-27
论文集名
2022 IEEE INTERNATIONAL CONFERENCE ON ELECTRICAL ENGINEERING, BIG DATA AND ALGORITHMS (EEBDA)
发表日期
2022
起止页码
525-530
DOI
10.1109/EEBDA53927.2022.9744732
收录
ISTP

作者

  • Y

    Yao, Shentao (第一作者)

  • L

    Li, Kun (通讯)

查看全部 6 位作者收起作者
  • X

    Xing, Kun

  • W

    Wu, Kewei

  • X

    Xie, Zhao

  • G

    Guo, Dan

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417