Multi-Scale Convolutional Neural Network-Based Intra Prediction for Video Coding

In both H.264/AVC and HEVC, the angular prediction is adopted for intra coding, which only exploits the spatial correlation between the current block and its neighboring single line reference. This angular prediction can handle the main directional patterns well, however, lacks the ability to deal with other directions. In this paper, a multi-scale convolutional neural network based intra prediction is proposed to address this problem. Specifically, a predicted block is first generated by the angular prediction, then fed into the proposed network with its neighboring reconstructed $L$ -shape to generate a more accurate predicted block. On one hand, the $L$ -shape of multiple lines provides more reliable reconstructed pixels and more contextual information to get better prediction; on the other hand, the multi-scale feature extraction takes the advantage of the feature maps in different scales to further enhance the prediction. With this multi-scale structure, the $L$ -shape can be used to refine both left-above and right-bottom pixels in the predicted block during the convolution operation. Experimental results demonstrate that compared with HEVC reference software HM 16.9, the proposed intra prediction can achieve an average of 3.4% (up to 5.6%) bitrate saving with all intra configuration.