Learning deep neural network based kernel functions for small sample size classification

Kernel learning is to learn a kernel function based on the set of all sample pairs from training data. Even for small sample size classification tasks, the set size is mostly large enough to make a complex kernel that holds lots of parameters being well optimized. Hence, the complex kernel can be helpful in improving classification performance via providing more meaningful feature representation in kernel induced feature space. In this paper, we propose to embed a deep neural network (DNN) into kernel functions, taking its output as kernel parameter to adjust the feature representations adaptively. Two kind of DNN based kernels are defined, and both of them are proved to satisfy the Mercer theorem. Considering the connection between kernel and classifier, we optimize the proposed DNN based kernels by exploiting the GMKL alternating optimization framework. A stochastic gradient descent (SGD) based algorithm is also proposed, which still implements alternating optimization in each iteration. Furthermore, an incremental batch size method is given to reduce gradient noise gradually in optimization process. Experimental results show that our method performed better than the typical methods.