Image Captioning With Controllable and Adaptive Length Levels

作者：Ding, Ning; Deng, Chaorui; Tan, Mingkui^*; Du, Qing; Ge, Zhiwei; Wu, Qi

来源：IEEE Transactions on Pattern Analysis and  Machine Intelligence, 2024, 46(2): 764-779.

DOI：10.1109/TPAMI.2023.3328298

• Length-controllable image captioning
• non-autoregressive image captioning
• length level reranking
• refinement-enhanced sequence training

Image captioning is a core challenge in computer vision, attracting significant attention. Traditional methods prioritize caption quality, often overlooking style control. Our research enhances method controllability, enabling descriptions of varying detail. By integrating a length level embedding into current models, they can produce detailed or concise captions, increasing diversity. We introduce a length-level reranking transformer to correlate image and text complexity, optimizing caption length for informativeness without redundancy. Additionally, with caption length increase, computational complexity grows due to the autoregressive (AR) design of existing methods. To address this, our non-autoregressive (NAR) model maintains constant complexity regardless of caption length. We've developed a training approach that includes refinement sequence training and sequence-level knowledge distillation to close the performance gap between NAR and AR models. In testing, our models set new standards for caption quality on the MS COCO dataset and offer enhanced controllability and diversity. Our NAR model excels over AR models in these aspects and shows greater efficiency with longer captions. With advanced training techniques, our NAR's caption quality rivals that of leading AR models.