Modeling model load states in multimodal learning: a cognitive load theory inspired framework for balanced multimodal learning

Multimodal learning aims to exploit cross-modal complementarity by integrating heterogeneous modalities such as text, audio, and vision. However, multimodal models often suffer from the imbalanced multimodal learning problem, where the dominant modality monopolizes optimization resources while weaker modalities remain insufficiently learned. Empirical studies show that, unlike unimodal training, gradients across modalities exhibit highly similar evolution trends in multimodal settings, even with existing balancing strategies, indicating that gradient magnitude adjustment alone fails to address the underlying optimization dynamics. In this work, we reanalyze imbalanced multimodal learning from an optimization perspective by incorporating Cognitive Load Theory. We show that, under limited model capacity, jointly optimizing multiple modalities induces a persistent high load state, where gradients from different modalities compete within a constrained update subspace, resulting in gradient conflicts and a persistent imbalanced state. Inspired by human cognitive behavior, we propose a multimodal balanced learning framework that integrates a plug-and-play multimodal alternating learning strategy to alleviate the learning load of simultaneous multimodal optimization and a Gaussian noise based optimizer to prevent overly similar gradient magnitude change trends. Extensive experiments on IEMOCAP, CMU-MOSEI, and AVE demonstrate the rationality and effectiveness of the proposed framework.