PENCIL: Prototype-ENhanced ComposItional Learning for Class-Incremental Hand Gesture Recognition

Gesture recognition systems, as an indispenable component of virtual reality technology, are typically required to continuously acquire and update knowledge in order to meet users demands for recognizing both historical and newlyregistered gestures in real-world applications. In this paper, we propose a novel yet effective Prototype-ENhanced ComposItional Learning (PENCIL) method for Class-Incremental Hand Gesture Recognition (CI-HGR) task. It attempt to achieve exemplarfree class-incremental learning through adaptive isolated parameter composition and fine-tuning. Specifically, the Task-Specific Compositional Mechanism (TCM) module in PENCIL designs the trainable indexable embedding for each sequentially-arrived tasks containing new gestures, enabling HGR model to dynamically composite different parameters based on diverse gesture inputs. Furthermore, to address the recognition confusion caused by strong similarities among different gestures representations, PENCIL involves the second module Prototype-Enhanced Hybrid Regularization (PHR). It leverage prototype pseudo samples to maintain the accuracy and stability of parameter composition in TCM. Extensive experiments demonstrate the efficacy and modelagnostic capability of our method. Moreover, PENCIL fulfils state-of-the-art performance, with remarkable accuracy boosts ( 13.33% and 22.43%) and overhead reduction (over 90%) in contrast to most competitive method.