Formation Tracking of Mobile Robots Using Only Constrained and Faulty Visual Feedback

This paper investigates the leader-follower formation tracking problem for nonholonomic mobile robots equipped solely with an RGB-D camera, eliminating the need for communication devices, which significantly reduces system complexity and costs. However, achieving robust tracking with this minimal configuration is highly challenging, as the follower's camera often fails to detect the leader due to two key issues that are often overlooked in existing research. First, the limited field-of-view (FOV) of the camera frequently allows the leader to move out of sight. Second, visual detection faults—including obstacle occlusion, lighting variations, depth image failures, and false detections caused by similar objects—complicate reliable tracking. To address these challenges, we propose a novel visual formation tracking scheme. Our approach allows the follower robot to maintain continuous tracking of the leader within its constrained FOV in the absence of visual faults, even without access to the leader's depth or velocity. When visual faults occur, the system swiftly diagnoses the fault type and resumes tracking accordingly. Our method leverages formation kinematics to enhance efficiency and reduce computation while requiring only an onboard camera, and its effectiveness is validated through simulations and real-world experiments.