ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
期刊论文

Towards Bridged Vision and Language: Learning Cross-Modal Knowledge Representation for Relation Extraction

Feng, Junhao; Wang, Guohua; Zheng, Changmeng*; Cai, Yi; Fu, Ze; Wang, Yaowei; Wei, Xiao-Yong; Li, Qing四川大学
IEEE Transactions on Circuits and Systems for Video Technology2024-1DOI: 10.1109/TCSVT.2023.3284474
6阅读
引用收藏
分享

摘要

In natural language processing, relation extraction (RE) is to detect and classify the semantic relationship of two given entities within a sentence. Previous RE methods consider only the textual contents and suffer performance decline in social media when texts lack contexts. Incorporating text-related visual information can supplement the missing semantics for relation extraction in social media posts. However, textual relations are usually abstract and of high-level semantics, which causes the semantic gap between visual contents and textual expressions. In this paper, we propose RECK - a neural network for relation extraction with cross-modal knowledge representations. Different from previous multimodal methods training a common subspace for all modalities, we bridge the semantic gaps by explicitly selecting knowledge paths from external knowledge through the cross-modal object-entity pairs. We further extend the paths into a knowledge graph, and adopt a graph attention network to capture the multi-grained relevant concepts which can provide higher level and key semantics information from external knowledge. Besides, we employ a cross-modal attention mechanism to align and fuse the multimodal information. Experimental results on a multimodal RE dataset show that our model achieves new state-of-the-art performance with knowledge evidence.

学科与关键词

关键词
Multimodal relation extractiongraph attention networkknowledge graphs
相关
加载中…
  • Multimodal Representation with Embedded Visual Guiding Objects for Named Entity Recognition in Social Media Posts
    Wu, Zhiwei*; Zheng, Changmeng*; Cai, Yi; Chen, Junying; Leung, Ho-fung; Li, Qing28th ACM International Conference on Multimedia (MM), ELECTR NETWORK, 2020-10-12 To 2020-10-16.被引 19
  • Object-Aware Multimodal Named Entity Recognition in Social Media Posts With Adversarial Learning
    Zheng, Changmeng; Wu, Zhiwei; Wang, Tao; Cai, Yi*; Li, QingIEEE TRANSACTIONS ON MULTIMEDIA, 2021, 23: 2520-2532.被引 321
  • DRAKE: Deep Pair-Wise Relation Alignment for Knowledge-Enhanced Multimodal Scene Graph Generation in Social Media Posts
    Fu, Ze; Zheng, Changmeng; Feng, Junhao; Cai, Yi*; Wei, Xiao-Yong; Wang, Yaowei; Li, QingIEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(7): 3199-3213.
查看全部 →

发表信息

论文状态
公开发表
期刊
IEEE Transactions on Circuits and Systems for Video Technology
发表日期
2024-01
卷期
第 34 卷, 第 1 期
页码
561-575
DOI
10.1109/TCSVT.2023.3284474
语言
ENGLISH
成果收录
SCIE
资助标注
National Natural Science Foundation of China

作者

  • F

    Feng, Junhao (第一作者)

  • W

    Wang, Guohua

查看全部 8 位作者收起作者
  • Z

    Zheng, Changmeng (通讯)

    hong kong polytech univ

  • C

    Cai, Yi

  • F

    Fu, Ze

  • W

    Wang, Yaowei

  • W

    Wei, Xiao-Yong

    sichuan univ

  • L

    Li, Qing

    hong kong polytech univ

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417