ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
期刊论文

MATR: Multimodal Medical Image Fusion via Multiscale Adaptive Transformer

Tang, Wei; He, Fazhi*; Liu, Yu; Duan, Yansong武汉大学
IEEE Transactions on Image Processing2022DOI: 10.1109/TIP.2022.3193288
58被引6阅读
引用收藏
分享

摘要

Owing to the limitations of imaging sensors, it is challenging to obtain a medical image that simultaneously contains functional metabolic information and structural tissue details. Multimodal medical image fusion, an effective way to merge the complementary information in different modalities, has become a significant technique to facilitate clinical diagnosis and surgical navigation. With powerful feature representation ability, deep learning (DL)-based methods have improved such fusion results but still have not achieved satisfactory performance. Specifically, existing DL-based methods generally depend on convolutional operations, which can well extract local patterns but have limited capability in preserving global context information. To compensate for this defect and achieve accurate fusion, we propose a novel unsupervised method to fuse multimodal medical images via a multiscale adaptive Transformer termed MATR. In the proposed method, instead of directly employing vanilla convolution, we introduce an adaptive convolution for adaptively modulating the convolutional kernel based on the global complementary context. To further model long-range dependencies, an adaptive Transformer is employed to enhance the global semantic extraction capability. Our network architecture is designed in a multiscale fashion so that useful multimodal information can be adequately acquired from the perspective of different scales. Moreover, an objective function composed of a structural loss and a region mutual information loss is devised to construct constraints for information preservation at both the structural-level and the feature-level. Extensive experiments on a mainstream database demonstrate that the proposed method outperforms other representative and state-of-the-art methods in terms of both visual quality and quantitative evaluation. We also extend the proposed method to address other biomedical image fusion issues, and the pleasing fusion results illustrate that MATR has good generalization capability. The code of the proposed method is available at https://github.com/tthinking/MATR.

学科与关键词

关键词
TransformersImage fusionSingle photon emission computed tomographyMagnetic resonance imagingTransformsMedical diagnostic imagingTask analysisbiomedical imagetransformeradaptive convolutiondeep learning
相关
加载中…
  • Transformer-Based End-to-End Anatomical and Functional Image Fusion
    Zhang, Jing; Liu, Aiping*; Wang, Dan; Liu, Yu; Wang, Z. Jane; Chen, Xun*IEEE Transactions on Instrumentation and Measurement, 2022, 71: 5019711.被引 1
  • Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks
    Liu, Yu; Shi, Yu; Mu, Fuhao; Cheng, Juan; Li, Chang; Chen, Xun*IEEE Transactions on Instrumentation and Measurement, 2022, 71: 4006015.被引 16
  • Editorial: Multimodal brain image fusion: Methods, evaluations, and applications
    Liu, Yu*; Ma, Jiayi*; Zhang, Qiang*; Wei, Wei*; Chen, Xun*; Liu, Zheng*Frontiers in Neuroscience, 2023, 16: 1128938.
查看全部 →

发表信息

论文状态
公开发表
期刊
IEEE Transactions on Image Processing
发表日期
2022
卷期
第 31 卷
页码
5134-5149
DOI
10.1109/TIP.2022.3193288
语言
ENGLISH
期刊等级
1区
成果收录
SCIE
资助标注
National Natural Science Foundation of China [62072348, 62176081]; Science and Technology Major Project of Hubei Province (Next-Generation Artificial Intelligence (AI) Technologies) [2019AEA170]; National Key Research and Development Program of China [2019YFC1509604]

作者

  • T

    Tang, Wei (第一作者)

    wuhan univ

  • H

    He, Fazhi (通讯)

    wuhan univ

查看全部 4 位作者收起作者
  • L

    Liu, Yu

  • D

    Duan, Yansong

    wuhan univ

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417