ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
期刊论文

Multimodal Image Fusion via Self-Supervised Transformer

Zhang, Jing; Liu, Yu; Liu, Aiping; Xie, Qingguo; Ward, Rabab; Wang, Z. Jane; Chen, Xun*1
IEEE SENSORS JOURNAL2023-5-1DOI: 10.1109/JSEN.2023.3263336
1被引7阅读
引用收藏
分享

摘要

Multimodal image fusion (MMIF) aims to synthesize one more informative image from complementary multimodal images. Due to the absence of ground truth, most MMIF works utilize unsupervised loss functions to preserve specific source information in the fused image, suffering from subjective information preservation and high demand for a large training set. Self-supervised learning is a feasible technology to avoid these issues. Current self-supervised MMIF works pretrain the network using a large natural image set via the vanilla image reconstruction task, then implement image fusion by integrating fusion rules into the well-trained network. However, they ignore the domain discrepancy between training data (i.e., the natural image set) and test data (i.e., the MMIF set). In addition, existing MMIF methods are mostly built based on traditional convolutional neural networks, leading to the inability of modeling long-range contextual relationships of source modalities. To this end, we propose a self-supervised transformer for MMIF. Under the self-supervised framework, we design random image super-resolution (RSR) as the pretext task to train the network. Both RSR and MMIF have similar multiresolution inputs and common goals: image restoration and enhancement, which are beneficial for diminishing the domain discrepancy between the train and test data. Moreover, we present a transformer-based fusion network for capturing both local and global information of source modalities. The proposed method is validated on multiple MMIF datasets. Qualitative and quantitative experimental results demonstrate that the proposed method can yield superior fusion quality compared to the state-of-the-art image fusion methods.

学科与关键词

关键词
TransformersTask analysisImage fusionFeature extractionTrainingSensorsImage reconstructionMultimodal image fusion (MMIF)self-supervised learningtransformer
相关
加载中…

文献加载失败,请重试

重试

发表信息

论文状态
公开发表
期刊
IEEE SENSORS JOURNAL
发表日期
2023-05-01
卷期
第 23 卷, 第 9 期
页码
9796-9807
DOI
10.1109/JSEN.2023.3263336
语言
ENGLISH
成果收录
SCIE
资助标注
Chinese Academy of Sciences (CAS) [YSBR-067]; National Natural Science Foundation of China [32271431, 82272070, 62176081, 61922075]; Fundamental Research Funds for the Central Universities [KY2100000123]

作者

  • Z

    Zhang, Jing (第一作者)

    univ sci & technol china

  • L

    Liu, Yu

查看全部 7 位作者收起作者
  • L

    Liu, Aiping

    univ sci & technol china

  • X

    Xie, Qingguo

    univ sci & technol china

  • W

    Ward, Rabab

    1

  • W

    Wang, Z. Jane

    1

  • C

    Chen, Xun (通讯)

    univ sci & technol china

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417