Uformer++: Light Uformer for Image Restoration

Based on UNet, numerous outstanding image restoration models have been developed, and Uformer is no exception. The exceptional restoration performance of Uformer is not only attributable to its novel modules but also to the network’s greater depth. Increased depth does not always lead to better performance, but it does increase the number of parameters and the training difficulty. In this paper, we propose Uformer++, a reconstructed Uformer based on an efficient ensemble of UNets of varying depths that partially share an encoder and co-learn simultaneously under deep supervision. Our proposed new architecture has significantly fewer parameters than the vanilla Uformer, but still with promising results achieved. Considering that different channel-wise features contain totally different weighted information and so are pixel-wise features, a novel Nonlinear Activation Free Feature Attention (NAFFA) module combining Simplified Channel Attention (SCA) and Simplified Pixel Attention (SPA) is added to the model. The experimental results on various challenging benchmarks demonstrate that Uformer++ has the least computational cost while maintaining performance.