MDNET: Multi-Scale Differential Transformer for Efficient Lightweight Image Classification

In recent years, deep neural networks have revolutionized image classification by enabling automatic feature learning. However, this performance gain comes at the cost of increased model complexity and computational demands. In many real-world application scenarios, constraints related to memory usage and response speed necessitate the development of efficient lightweight models. To address this, we propose MDNET, a highly efficient multi-scale differential lightweight network without additional model deep compression for image classification. MDNET incorporates a Multi-scale Differential Transformer (MDIFFormer) architecture that integrates diverse receptive fields along the channel dimension, rather than the spatial dimension, enabling parallel extraction of multi-scale information and facilitating global context understanding. A differential attention mechanism inspired by the differential amplifier circuit in physics is introduced to mitigate attention noise from irrelevant contexts, enhancing model robustness. Furthermore, label smoothing is applied to the probability distribution, introducing uniform noise to reduce over-reliance on true labels. Extensive experiments on benchmark datasets, including CIFAR100, CIFAR10, and a newly released dataset focused on new energy battery packs (NEBP), demonstrate that MDNET achieves state-of-the-art performance while maintaining low computational complexity. Here, we show that MDNET achieves a Top-1 accuracy of 73.34% on CIFAR100 with only 2.38M parameters, outperforming existing lightweight models.