FedSS: A Federated Semantic Segmentation Framework with Domain-Agnostic Feature Extraction and Fair Aggregation

Domain heterogeneity in federated learning presents significant challenges, particularly in complex tasks like semantic segmentation, where pixel-level accuracy is crucial. Existing approaches primarily focus on extracting domain-specific features, but they often overlook the importance of class-level feature invariance. In this paper, we propose FedSS, a novel federated semantic segmentation framework designed to address domain heterogeneity. FedSS introduces a Fine-grained Domain-agnostic Feature Extraction (DFE) module that standardizes feature maps using category-level statistics to ensure domain-agnostic feature extraction. Additionally, the Adaptive Ordering Based Feature Decorrelation (AFD) module enhances the model's ability to distinguish between different semantic categories by decorrelating feature channels. To further tackle domain discrepancies, we present a Domain Fairness-aware Aggregation Strategy (DFA) that dynamically adjusts aggregation weights based on local domain variations. Our approach improves the robustness and generalization of the semantic segmentation model across diverse domains, ensuring more accurate and reliable pixel-level predictions. Experimental results demonstrate the effectiveness of FedSS in addressing domain heterogeneity and enhancing segmentation performance in federated learning settings.The source code is released at https://github.com/vibratingwings/FedSemanSeg.