摘要
The latest methods for saliency prediction on 360 degrees images show that better results can be obtained using equirectangular (ERP) images as input. Due to the limitation of the receptive field, existing convolution-based networks cannot capture long-range information in complex 360 degrees images. Although the transformer has the innate ability to capture long-range correlations with self-attention, large dataset requirement limit its application in saliency prediction of 360 degrees images. In this paper, we present a novel Multi-scale Transformer framework for Saliency prediction on 360 degrees images (MTSal360). The Multi-scale Transformer Module (MTM) is designed in the network to aggregate the contextual long-range information, which includes a Convolutional Positional Encoder (CPE) to enable the model could train and test on cubic and ERP format separately to address the insufficient data. Experiments on two public datasets illustrate that MTSal360 achieves better results over the state-of-the-art methods.
学科与关键词
- SalBiNet360: Saliency Prediction on 360° Images with Local-Global Bifurcated Deep NetworkChen, Dongwen; Qing, Chunmei*; Xu, Xiangmin; Zhu, Huansheng27th IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), Atlanta, GA, 2020-03-22 To 2020-03-26.被引 7
- Attentive and context-aware deep network for saliency prediction on omni-directional imagesQing, Chunmei; Zhu, Huansheng; Xing, Xiaofen*; Chen, Dongwen; Jin, JianxiuDigital Signal Processing: A Review Journal , 2022, 120: 103289.被引 4
- Intra- and Inter-Reasoning Graph Convolutional Network for Saliency Prediction on 360° ImagesChen, Dongwen; Qing, Chunmei*; Lin, Xu; Ye, Mengtao; Xu, Xiangmin; Dickinson, PatrickIEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(12): 8730-8743.
