Knowledge distillation based on decision boundary instances generated by DBI-GAN

作者：Zhu, Ziqi; Liu, Xi; Deng, Chunhua; Liu, Jing^*; Zou, Jixin

来源：NEUROCOMPUTING, 2022, 487: 190-202.

DOI：10.1016/j.neucom.2021.10.097

• Knowledge distillation
• Decision boundary instance
• Generative adversarial network

In this paper, we propose a new model-free knowledge distillation scheme. We attempt on distilling the knowledge about the decision-making process and design a novel generative adversarial network, named as DBI-GAN, to generate instances located on the decision boundary of the teacher network. By depicting the decision-making process of the teacher network with the decision boundary instances, it is possible for us to transfer such knowledge to a student network. Based on the decision boundary instances, a new knowledge distillation scheme is proposed. In our scheme, we focus on the student network's "weak-points region", where wrong decisions are made. We locate these weak-points regions by evaluate the student network using confusion matrix. Then, a specific dataset, which consists of the original train data and synthesized DBIs that located in the weak-points regions, is constructed for knowledge transferring. With the constructed sample set, the student network is then optimized with a new KD loss function. We evaluate the effectiveness of proposed scheme on banchmark datasets and state-of-the-art performance is achieved.