Object-aware hybrid map for indoor robot visual semantic navigation

In order to achieve an intuitive interaction and visual semantic navigation for the indoor robot, we propose a novel object-aware hybrid map. The existing map is usually a metric map, lacking semantics for interaction. We combine objects in the indoor environment with the metric map to constitute a hybrid map. The map consists of a 3D object semantic map and a 2D occupancy grid map, which transfers human commands to the grid map through object semantics, thereby enabling autonomous navigation for the robot. We utilize ORB-SLAM2 for continuous pose estimation and 3D mapping. 2D object detection in key-frames is conducted based on YOLO v3. The object point clouds in multiple perspectives are merged and a 3D bounding box of the object is estimated. These objects construct a 3D semantic map. Furthermore, we project a 3D point cloud map into a 2D plane in order to get an occupancy grid map. Finally, these two maps are combined forming an object-aware hybrid map. We conduct experiments in real environments in order to verify the feasibility and robustness of the hybrid map for robot semantic navigation.