Focalized contrastive view-invariant learning for self-supervised skeleton-based action recognition

作者：Men, Qianhui^*; Ho, Edmond S. L.; Shum, Hubert P. H.; Leung, Howard

来源：NEUROCOMPUTING, 2023, 537: 198-209.

DOI：10.1016/j.neucom.2023.03.070

• Self -supervised learning
• Skeleton -based action recognition
• Contrastive learning

Learning view-invariant representation is a key to improving feature discrimination power for skeleton -based action recognition. Existing approaches cannot effectively remove the impact of viewpoint due to the implicit view-dependent representations. In this work, we propose a self-supervised framework called Focalized Contrastive View-invariant Learning (FoCoViL), which significantly suppresses the view-specific information on the representation space where the viewpoints are coarsely aligned. By maximizing mutual information with an effective contrastive loss between multi-view sample pairs, FoCoViL associates actions with common view-invariant properties and simultaneously separates the dissimilar ones. We further propose an adaptive focalization method based on pairwise similarity to enhance contrastive learning for a clearer cluster boundary in the learned space. Different from many existing self-supervised representation learning work that rely heavily on supervised classifiers, FoCoViL performs well on both unsupervised and supervised classifiers with superior recognition perfor-mance. Extensive experiments also show that the proposed contrastive-based focalization generates a more discriminative latent representation.