Robot learning from demonstration for assembly with camera-supplemented optical motion capture sensors

Learning from demonstration (LfD) has been exhibiting its tremendous potential in transferring human worker’s experience to robots by demonstrating targeted tasks, especially in manufacturing industries. Motion capture (MoCap) sensors are essential to collect human demonstration data in robot LfD systems. Marker-based sensors, typically optical MoCap sensors, possess high motion capture accuracy and they are usually selected for precision assembly motion capture. However, precise and small-range movements are challenging to be recognized and distinguished through the pose trajectory data obtained by the marker-based sensor itself. Those actions can be recognized by marker-less sensors, typically video cameras with neural network learning techniques. In this paper, we propose a camera-supplemented optical motion capture sensors system by introducing an RGB camera marker-less sensor to facilitate the optical MoCap sensors to recognize action types. A robot assembly LfD platform is thus presented with the novel camera-supplemented optical motion capture sensors system. The 6-dimensional pose of parts and video of the human workers’ assembly demonstration are simultaneously obtained and temporally aligned. Different Assembly Movement Primitives are defined and their corresponding policies are designed, consisting of a Cartesian impedance controller and goal pose update law. An action recognition network and a policy parameter estimation method are proposed to generate assembly strategies that can be learnt by robots, which exploit both the accuracy of the pose trajectory data and the informational abundance of the demonstration video data. Robotic assembly experiments were conducted to validate the proposed demonstration platform and policy learning method. Experiment results indicate that, with the proposed demonstration platform and the policy learning method, the robot can efficiently accomplish precise assembly tasks with a high success rate.