FedSea: Federated Learning via Selective Feature Alignment for Non-IID Multimodal Data

The growing demands for privacy protection challenge the joint training of one model by leveraging multiple datasets. Federated learning (FL) provides a new way to overcome this challenge and has attracted many research interests, which enables multiple parties to collaboratively train a machine learning model without exchanging their local data. Despite some success, the non-independent and identically distributed (non-IID) data distributions in different parties remain challenging and easily damage the performance of FL methods, specifically for the heterogeneous multimodal data. Existing FL studies on non-IID data settings are often dedicated to the label space, neglecting the non-IID issues in feature space, thus limiting their performance when the parties with non-IID multimodal data. This paper proposes a new Federated learning method via Selective feature Alignment (FedSea) to align representations across multiple parties in the feature space. FedSea uses a domain adversarial learning framework consisting of an affine-transform-based generator and a gradient-reversal-based client discriminator to perform IID transformation and reduce data source distinguishability, respectively. An attention-based mask module and a feature IID confidence quantification method are introduced to effectively address the diverse feature non-IID levels across multimodal data. Comprehensive experiments are conducted on three widely-used public datasets and one large-scale industrial dataset, showing FedSea has: 1) better performance than state-of-the-art FL methods on both multimodal and single-modal datasets; 2) superior feature alignment ability on non-IID datasets, and 3) good model interpretability.