Fusion decomposition and backbone gathering based multimodal sentiment analysis under uncertain missing modalities

Recently, Multimodal Sentiment Analysis (MSA) under uncertain missing modalities has become a critical challenge in the field of emotional computing. Although some effective models have been developed, these models largely fail to consider the influence of modalities’ quality on the multimodal fusion and missing modalities’ completion. To tackle these issues, we propose a Fusion Decomposition and Backbone Gathering based MSA model under uncertain missing modalities, FDBG, which includes four innovative works. (1) Inspired by the pyramid convolution structure, we devise a pyramid multi-head attention mechanism, which serves as an important calculation foundation for the feature learning in FDBG; (2) To improve the quality of each modality, we suggest a pyramid prior enhancement method that combines the current sample with several similar historical samples; (3) We put forward a fusion decomposition method to effectively complete the missing modalities, which first fuses the three modalities with the pyramid self-attention, and then decomposes the fusion feature into three modalities and completes the missing modalities with the self-attention interaction mechanism; and (4) To eliminate the interference, we introduce a backbone gathering method to translate and fuse the three modalities into a single modality. Based on three public benchmark datasets (IEMOCAP, MELD and CMU-MOSI), we conduct extensive experiments and prove that FDBG outperforms the 12 baseline models. The source codes of this work are available at https://github.com/SHX-AI/FDBGMSA.