Multi-task mutual learning for multimodal emotion-cause pair extraction in conversations

Multimodal Emotion-Cause Pair Extraction (ECPE) in Conversations (MC-ECPE) aims to simultaneously identify emotions and their causes within conversations across different modalities. Early paradigms of ECPE involved a two-step framework for emotion and cause extraction and pairing, resulting in error accumulation. Thus, there is a growing interest in end-to-end ECPE. Despite the progress, emotion, and cause extraction are essentially mutually dependent, yet existing efforts fail to model it deeply. Additionally, Baselines for the MC-ECPE task primarily use traditional fusion methods like concatenation, limiting context understanding between modalities. Based on this, we propose the Multi-Task Mutual Learning (MTML) framework, which utilizes implicit and explicit modeling strategies to model the mutual dependency between emotion and cause extraction. Specifically, we introduce a Multimodal Interactive Graph Attention Network (MIGAT) with three types of connections: intra-modal for conversational context, cross-modal for multimodal fusion, and cross-task for capturing dependencies between emotion and cause extraction tasks. Implicit modeling leverages cross-task connections within MIGAT and information from shared components in progressive multi-task learning (PMTL), while explicit modeling iteratively extracts emotional and causal probability distributions to enhance subsequent reasoning. Experimental results demonstrate the superiority of our MTML over state-of-the-art methods.