Real-time Stereo Matching Networks based on Multi-modal and Sparse-dense Fusion

Binocular depth estimation models using deep learning struggle to balance accuracy and efficiency, consuming significant memory during inference. We present a novel, concise, and scalable model architecture, Fusion-Stereo, based on disparity iterative optimization. It ensures excellent prediction accuracy for stereo vision sensors while achieving real-time inference speed and low memory usage. Observing the different modal information of image texture, matching similarity, and disparity features, we propose a new disparity residual prediction module for adaptive propagation of reliable disparity information in spatial and disparity dimensions. To enhance the reliability of depth prediction, an efficient fusion mechanism is designed between Fusion-Stereo and external prior data sources. Sparse disparity priors are introduced to the model to assist dense disparity prediction, maintaining essentially unchanged inference speed and memory occupation.