Panoramic Video Inter Frame Prediction and Viewport Prediction Based on Background Modeling

While panoramic videos provide people with a realistic and immersive viewing experience through a 360° field of view (FoV) and adjustable viewports, the wide FoV results in large data volume, which puts great pressure on video storage and transmission. To address this issue, numerous panoramic video compression methods based on eliminating spatial redundancy have been developed. However, none of these methods consider the background redundancy observed in various panoramic videos applications like surveillance and game live streaming. In this paper, we propose a background modeling-based panoramic video compression approach to improve user experience in scenarios where background changes are not obvious. Specifically, we study two background modeling schemes and utilize the constructed background as a long-term reference frame in SVT-HEVC coding framework. Besides, we develop a viewport prediction approach by combining the constructed background with edge information. Experimental results show that the proposed method can achieve a gain of about 8% compared to the original encoder, and enhances the smoothness and clarity of the reconstructed panoramic videos. Compared with existing deep learning-based viewport prediction methods, our method only takes half the time to predict the viewport with essentially the same accuracy.