TFST-Net: A Time-Frequency and Spatio-Temporal Attention Network for Hybrid EEG-fNIRS Brain-Computer Interfaces

Hybrid brain-computer interface (BCI) systems combining electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have garnered significant interest due to their potential to overcome the limitations of EEG-or fNIRS-based systems. However, most hybrid EEG-fNIRS BCIs face challenges in effectively capturing the joint spatial-temporal-frequency dynamic information from EEG and fNIRS signals. To address this issue, we propose a time-frequency and spatio-temporal attention network (TFST-Net). The TFST-Net integrates EEG and fNIRS signals through three distinct pathways to enhance the representation of brain neural activity and hemodynamic states. The EEG signals are processed using the proposed time-frequency attention mechanism, while a spatio-temporal attention mechanism is proposed to capture the spatial and temporal dynamics of hemodynamic responses in fNIRS signals. The features from EEG, deoxygenated hemoglobin (HbR), and oxygenated hemoglobin (HbO) are fused to generate the final output, effectively leveraging the unique strengths of each modality and capturing the joint information for improved performance. Experimental results demonstrate that TFST-Net outperforms existing methods, providing a promising approach to hybrid EEG-fNIRS BCI applications.