HTMP: Triple-Scale Multilevel Network With Hessian Spatial Loss for Pansharpening

Pansharpening aims to recover the spectral information and spatial details of high-resolution multispectral (HRMS) images with high accuracy. In this context, we design a triple-scale multilevel network guided by Hessian spatial loss for pansharpening (HTMP). First, we propose a novel Hessian spatial loss designed to establish deep mapping relationships in the spatial domain. Hessian spatial loss guides the network in enhancing its ability to characterize the edges of blurred regions while maintaining the consistency of spatial details. Second, we employ a triple-scale multilevel feature extraction network (TMFENet) to obtain comprehensive spectral and spatial features, thereby enhancing the interaction of multiscale contextual information from coarse-grained to fine-grained scales. To efficiently integrate and represent cross-modal long-distance information, the spectral and spatial features are treated as a whole and input into the triple-scale adaptive sparse Transformer (TASTrans) to extract global features. Finally, the multilevel image reconstruction network (MIRNet) combines multimodal features at different resolutions to progressively generate HRMS images rich in semantics. Experiments performed on datasets demonstrate that our method produces fused images with superior visual quality compared to state-of-the-art methods. Furthermore, it is obvious from the ablation experiments that incorporating the Hessian spatial loss significantly enhances the fusion performance of deep learning (DL) models.