IV-Net: single-view 3D volume reconstruction by fusing features of image and recovered volume

作者：Sun, Beibei; Jiang, Ping^*; Kong, Dali; Shen, Ting

来源：VISUAL COMPUTER, 2023, 39(12): 6237-6247.

DOI：10.1007/s00371-022-02725-6

• Single-view 3D reconstruction
• Multi-scale convolution
• Deep learning
• Residual convolutional neural network

Single-view 3D reconstruction aims to recover the 3D shape from one image of an object and has attracted increasingly attention in recent years. Mostly, previous works are devoted to learning a mapping from 2 to 3D, and lack of spatial information of objects will cause inaccurate reconstruction on the details of objects. To address this issue, for single-view 3D reconstruction, we propose a novel voxel-based network by fusing features of image and recovered volume, named IV-Net. By a pre-trained baseline, it achieves image feature and a coarse volume from each image input, where the recovered volume contains spatial semantic information. Specially, the multi-scale convolutional block is designed to improve 2D encoder by extracting multi-scale image information. To recover more accurate shape and details of the object, an IV refiner is further used to reconstruct the final volume. We conduct experimental evaluations on both synthetic ShapeNet dataset and real-world Pix3D dataset, and results of comparative experiments indicate that our IV-Net outperforms state-of-the-art approaches about accuracy and parameters.