Augmented reality-based virtual-real fusion for assembly guidance system

Aiming at the need for real-time part tracking, accurate registration, and effective virtual-real occlusion handling in augmented-reality-assisted manual assembly of complex products, this study presents a deployable markerless augmented reality assembly guidance framework that integrates lightweight keypoint-based initialization, contour-based real-time tracking, large-displacement correction, and depth-assisted occlusion handling. Specifically, the framework uses a neural network with multi-scale feature extraction to obtain sub-pixel keypoints for high-precision initial pose estimation of the assembly object. Based on this initialization, a contour-point tracking and pose transformation model is constructed to estimate the six-degree-of-freedom pose of the object, and a large-displacement correction module is introduced to maintain robust tracking in dynamic assembly environments. The proposed system is implemented and tested on a HoloLens 2-based manual assembly scenario, where its real-time tracking performance and virtual-real occlusion handling are evaluated. Experimental results show that the system achieves a tracking frame rate of 30 fps in the manual operation range, with a maximum tracking error below 2.5 mm and 2°. The results demonstrate that the proposed framework can effectively support assembly guidance visualization and satisfy the requirements of real-time, stable target tracking and virtual-real occlusion rendering.