Reinforcement Learning Strategy Based on Multimodal Representations for High-Precision Assembly Tasks

Robotic peg-in-hole task has always attracted researchers’ attention. With the development of real-time sensors and machine learning algorithms, collaborative robots are now having potential to insert tiny and delicate components of digital products. Due to grasping error, the absolute position of the peg would not be calculated directly by forward kinematics, but through high-resolution sensors. However, for each single modality, such as RGB-D image and proprioception, has its own limitation during the insertion process. Camera cannot provide accurate information when the peg is closed to the target, while force/torque sensor is entirely blind before contact status begin. This paper used multimodal fusion method to utilize all the valuable information from multiple sensors. Representation cores from multimodal data were trained to forecast relative position between the peg and hole. Reinforcement learning network was then able to use the relative position to generate appropriate action of the robot. This paper verified the above algorithms through USB-C insertion experiments in ROS-Gazebo simulation.