A fractional-order momentum optimization approach of deep neural networks

The development of universal and high-efficiency optimization algorithms is a very important research direction of neural networks. Stochastic Gradient Decent Momentum(SGDM) is one of the most successful optimization algorithms, and easily fall into local extremes minimum. Inspired by the prominent success of Fractional-order Calculus in automatic control, we proposed a method based on Fractional-Order named Fractional-Order Momentum(FracM). As a natural extension of integral calculus, fractional order calculus inherits almost all the characteristics of integral calculus, and have some memorization and nonlocality. FracM performs fractional-order difference of momentum and gradient in SGDM algorithm. FracM can partially solve the problem of traps in the local minimum point and accelerated the train process. The proposed FracM optimization method can compare with the most advanced SGDM and Adam and other advanced optimization algorithm in terms of classification accuracy. The experiments show that FracM outperforms other optimizers on CIFAR10/100 and textual datasets IMDB with transformer-based models.