摘要
Visual Question Answering (VQA) requires reasoning about the visually-grounded relations in the image and question context. A crucial aspect of solving complex questions is reliable multi-hop reasoning, i.e., dynamically learning the interplay between visual entities in each step. In this paper, we investigate the potential of the reasoning graph network on multi-hop reasoning questions, especially over 3 "hops."We call this model QMRGT: A Question-Guided Multi-hop Reasoning Graph Network. It constructs a cross-modal interaction module (CIM) and a multi-hop reasoning graph network (MRGT) and infers an answer by dynamically updating the inter-associated instruction between two modalities. Our graph reasoning module can apply to any multi-modal model. The experiments on VQA 2.0 and GQA (in fully supervised and O.O.D settings) datasets show that both QMRGT and pre-training V&L models+MRGT lead to improvement on visual question answering tasks. Graph-based multi-hop reasoning provides an effective signal for the visual question answering challenge, both for the O.O.D and high-level reasoning questions.
学科与关键词
- Graph-Based Multi-Interaction Network for Video Question AnsweringGu, Mao; Zhao, Zhou*; Jin, Weike; Hong, Richang; Wu, FeiIEEE Transactions on Image Processing, 2021, 30: 2758-2770.被引 29
- Multi-Modal Alignment of Visual Question Answering Based on Multi-Hop Attention MechanismXia, Qihao; Yu, Chao*; Hou, Yinong; Peng, Pingping; Zheng, Zhengqi; Chen, WenElectronics, 2022, 11(11): 1778.被引 4
- ClueReader: Heterogeneous Graph Attention Network for Multi-Hop Machine Reading ComprehensionGao, Peng*; Gao, Feng; Wang, Peng; Ni, Jian-Cheng; Wang, Fei; Fujita, HamidoElectronics, 2023, 12(14): 3183.
