Skip to main navigation Skip to search Skip to main content

RMSAGF: A synergistic generation and fusion framework for robust multimodal sentiment analysis

  • Heilongjiang University
  • Zhejiang University
  • Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
  • School of Physics, Harbin Institute of Technology

Research output: Contribution to journalArticlepeer-review

Abstract

Multimodal Sentiment Analysis (MSA) with uncertain missing modalities poses a new challenge in sentiment analysis. To address this problem, existing methods often suffer from either ineffective fusion strategies for complex interactions or a lack of robustness when modality data is incomplete. In this paper, we propose a novel framework, RMSAGF, which features a synergistic dual-component architecture designed to address these issues systematically. First, for complete data, an Interdependent Multimodal Attention (IMA) module is designed for deep, iterative fusion to capture comprehensive inter- and intra-modal dynamics. Second, to handle missing modalities, a Context-Aware Feature Generation (CFG) module introduces an active generation paradigm. When a modality is missing, the CFG module first dynamically assesses the reliability of the present modalities to guide a dual-level generation strategy. It performs rapid utterance-level compensation while simultaneously generating a complete, temporally coherent feature sequence for the missing modality via a context-aware Transformer decoder. Extensive experiments on the CMU-MOSI, CMU-MOSEI, and CH-SIMS benchmarks demonstrate that RMSAGF establishes a new state of the art, particularly on core regression metrics under incomplete settings, thereby validating the superiority of its active generation paradigm.

Original languageEnglish
Article number131841
JournalExpert Systems with Applications
Volume316
DOIs
StateAccepted/In press - 2026
Externally publishedYes

Keywords

  • Context-aware generation
  • Cross-attention
  • Dynamic adaptation
  • Interdependent attention
  • Multimodal sentiment analysis
  • Transformer decoders

Fingerprint

Dive into the research topics of 'RMSAGF: A synergistic generation and fusion framework for robust multimodal sentiment analysis'. Together they form a unique fingerprint.

Cite this