Rethinking Neuromorphic Object Detection with Hybrid Dynamic Interaction Transformers

Combining the complementary benefits of frames and events has become a widely adopted strategy for object detection in challenging scenarios, such as low-light conditions and high-speed motion blur. However, most multimodal object detection methods employ two independent Artificial Neural Network (ANN) branches to process frames and events, which limits cross-modality information interaction between the two visual streams and poses challenges in extracting temporal cues from event streams with low power consumption. To address these challenges, we propose HDI-Former, a Hybrid Dynamic Interaction ANN-SNN Transformer, marking the first trial to design a directly trained hybrid ANN-SNN architecture for high-accuracy and energy-efficient object detection using asynchronous frames and events. Technically, we first present a semantic-enhanced self-attention mechanism for the Transformer architecture that explicitly models the relative semantic relationships between image encoding tokens to strengthen the correlation between them. Then, we design a Spiking Neural Network (SNN) branch, namely Spiking Swin Transformer, to extract temporal cues from event streams with low power consumption. Finally, we propose a bio-inspired dynamic interaction mechanism between ANN and SNN sub-networks to align the asynchronous modalities and achieve cross-modality information interaction. The results show that our HDI-Former outperforms state-of-the-art methods and our baselines by a large margin. Our SNN branch achieves comparable performance to the ANN with the same architecture while consuming 10.57× and 11.46× less energy on the DSEC-Detection and PKU-DAVIS-SOD datasets, respectively. Our code is available at https://github.com/dianzl/hdiformer.