Enhancing Robustness of Hand Gesture Recognition Against Sensor Data Loss by Fusing High-Density sEMG and Kinematics

The seamless integration of humans with digital environments hinges on robust and intuitive control interfaces, making accurate hand gesture recognition crucial for effective human-computer interaction (HCI). However, systems relying solely on kinematic sensors, such as data gloves, often exhibit a sharp performance decline when faced with partial data loss–a common occurrence in real-world scenarios that undermines their practical reliability. To address this robustness challenge, this study proposes and evaluates a multi-modal fusion framework leveraging Convolutional Neural Networks (CNNs). The framework integrates 64-channel High-Density surface Electromyography (HD-sEMG), which captures underlying motor intent, with kinematic information (finger joint angles and wrist posture) from a data glove. We classify a challenging set of 12 fine-grained gestures, composed of 4 distinct finger pinch types combined with 3 wrist postures. Experimental results demonstrate that under ideal, complete data conditions, the proposed multi-modal model achieves a high average classification accuracy of 98.4%, modestly outperforming single-modality counterparts. More importantly, a robustness evaluation simulating a 30% random loss of finger angle sensor channels revealed that while the accuracy of the kinematics-only model plummeted to 52.62%, our multi-modal fusion model maintained a significantly higher accuracy of 83.83%. These findings quantitatively confirm that the fusion of HD-sEMG with kinematic data provides a critical layer of redundancy, substantially enhancing the robustness of gesture recognition systems against incomplete or degraded sensor information. This work provides valuable insights for the development of highly reliable and practical HCI systems prepared for real-world operational uncertainties.