TCUP-Fusion: Transformer and Convolutional Neural Network based Ultrasound and Photoacoustic Image Fusion

Multimodal medical imaging provides a superior diagnostic profile compared to single-modality imaging, establishing the importance of multimodal medical image fusion (MMIF) for clinical diagnosis and therapeutic applications. Photoacoustic (PA) and ultrasound (US) imaging have become important diagnostic tools as they possess real-time noninvasive and radiation-free features. However, Traditional and emerging deep learning-based fusion techniques struggle to balance accuracy with real-time performance. We propose an end-to-end unsupervised fusion framework based on Transfomer and Convolutional Neural Networks (CNN) called TCUP-Fusion to solve these problems. The framework achieves efficient extraction of features by designing CNN and Biformer based encoders, while using the proposed SCCSA attention to guide the fusion of features, and finally achieves the representation of the fused image relying on a multi-feature fusion decoder. Finally, we conducted qualitative and quantitative comparison experiments on three different modal PA/US datasets. The results show that TCUPFusion outperforms other state-of-the-art fusion methods on US¹-PA fusion datasets. In addition, the algorithm in this paper also performs well on other modal image datasets (CT-MRI, MRI-PET), proving that has good robustness.