Computer Vision-Based Grasp Pattern Recognition with Application to Myoelectric Control of Dexterous Hand Prosthesis

Artificial intelligence provides new feasibilities to the control of dexterous prostheses. To achieve suitable grasps over various objects, a novel computer vision-based classification method assorting objects into different grasp patterns is proposed in this paper. This method can be applied in the autonomous control of the multi-fingered prosthetic hand, as it can help users rapidly complete 'reach-and-pick up' tasks on various daily objects with low demand on the myoelectric control. Firstly, an RGB-D image database (121 objects) was established according to four important grasp patterns (cylindrical, spherical, tripod, and lateral). The image samples in the RGB-D dataset were acquired on a large variety of daily objects of different sizes, shapes, postures (16), as well as different illumination conditions (4) and camera positions (4). Then, different inputs and structures of the discrimination model (multilayer CNN) were tested in terms of the classification success rate through cross-validation. Our results showed that depth data play an important role in grasp pattern recognition. The bimodal data (Gray-D) integrating both grayscale and depth information about the objects can improve the classification accuracy acquired from the RGB images (> 10%) effectively. Within the database, the network could achieve the classification with high accuracy (98%); it also has a strong generalization capability on novel samples (93.9 ± 3.0%). We finally applied the method on a dexterous prosthetic hand and tested the whole system on performing the 'reach-and-pick up' tasks. The experiments showed that the proposed computer vision-based myoelectric control method (Vision-EMG) could significantly improve the control effectiveness (6.4 s), with comparison to the traditional coding-based myoelectric control method (Coding-EMG, 13 {s} ).