摘要
Text-to-image synthesis refers to generating visual-realistic and semantically consistent images from given textual descriptions. Previous approaches generate an initial low-resolution image and then refine it to be high-resolution. Despite the remarkable progress, these methods are limited in fully utilizing the given texts and could generate text-mismatched images, especially when the text description is complex. We propose a novel fine-grained text-image fusion based generative adversarial networks (FF-GAN), which consists of two modules: Fine-grained text-image fusion block (FF-Block) and global semantic refinement (GSR). The proposed FF-Block integrates an attention block and several convolution layers to effectively fuse the fine-grained word-context features into the corresponding visual features, in which the text information is fully used to refine the initial image with more details. And the GSR is proposed to improve the global semantic consistency between linguistic and visual features during the refinement process. Extensive experiments on CUB-200 and COCO datasets demonstrate the superiority of FF-GAN over other state-of-the-art approaches in generating images with semantic consistency to the given texts.
学科与关键词
- Enhancing fine-detail image synthesis from text descriptions by text aggregation and connection fusion moduleZhou, Huaping; Wu, Tao; Ye, Senmao; Qin, Xinru; Sun, Kelei*Signal Processing: Image Communication , 2024, 122: 117099.
- CcGL-GAN: Criss-Cross Attention and Global-Local Discriminator Generative Adversarial Networks for text-to-image synthesisYe, Xihong; Lu, Lu*International Joint Conference on Neural Networks (IJCNN), ELECTR NETWORK, 2021-07-18 To 2021-07-22.
- GMF-GAN: Gradual multi-granularity semantic fusion GAN for text-to-image synthesisJin, Dehu; Li, Guangju; Yu, Qi; Yu, Lan; Cui, Jia*; Qi, Meng*Digital Signal Processing: A Review Journal , 2023, 140: 104105.
