Crossmodal Sequential Interaction Network for Hyperspectral and LiDAR Data Joint Classification

Numerous deep learning (DL) studies have indicated that fusing hyperspectral (HS) and light detection and ranging (LiDAR) data is effective for land-cover classification. However, the sequential characteristics (seqCHAs) in the spatial domain are always ambiguous and neglected. In this letter, we propose a deep crossmodal sequential interaction network (CsiNet) for HS and LiDAR data joint classification. We aim to verify the contributions of crossmodal seqCHAs in multimodal joint classification tasks and present an effective crossmodal sequential flattening (SF) strategy. Specifically, CsiNet sorts the neighboring samples in terms of the spectral and 3-D spatial diversities between the corresponding samples and the central one. Notably, the 3-D spatial diversity considers the shared sample positions in both modalities and the sample elevations in LiDAR data simultaneously. CsiNet is capable of extracting crossmodal sequential features comprehensively by long and short-term memory (LSTM) layers and better simulating sequential properties of samples compared with convolutional layer based networks. Experiments conducted on the Muufl Gulfport (MUUFL) and Houston 2013 datasets prove that CsiNet outperforms several state-of-the-art techniques qualitatively and quantitatively. When using 1% training samples per category, the overall accuracies of CsiNet on both datasets achieve 90.27% and 92.41% and are increased by 0.27% and 0.17% than the best comparison technique, respectively. Ablation experiments verify the effectiveness of CsiNet by replacing the crossmodal SF strategy with several alternative ones. All codes are available at https://github.com/GEOywb/CsiNet.