Temporality-guided Masked Image Consistency for Domain Adaptive Video Segmentation

Video semantic segmentation has witnessed substantial advancements, largely due to the vast volume of labeled training samples. Nevertheless, domain adaptive video segmentation that adapts from a labeled source domain to an unlabeled target domain remains insufficiently delved into. In this paper, we propose Temporality-guided Masked Image Consistency (TgMIC), a simple yet effective approach that leverages the concept of Masked Image Modeling (MIM) to learn semantic features in the target domain. Unlike random masking strategy applied in traditional MIM, TgMIC introduces a novel temporality-guided masking strategy that samples the mask according to the distribution of optical flow, which facilitate the learning of spatial context relations in video sequence. Specifically, TgMIC masks the patches in vision transformers where the variance of optical flow is large, as these patches are known to contain noisy estimates of optical flow. In order to learn semantic information for video segmentation, TgMIC reconstructs the predictions of original frames from the masked frames. Comprehensive tests and detailed analysis on various public datasets show that our mechanism stands out, outpacing contemporaneous techniques, while ensuring no additional time costs.