CovLets: A second-order descriptor for modeling multiple features

State-of-the-art techniques for image and video classification take a bottom-up approach where local features are aggregated into a global final representation. Existing frameworks (i.e., bag of words or Fisher vectors) are specifically designed to aggregate vector-valued features such as SIFT descriptors. In this article, we propose a technique to aggregate local descriptors in the form of covariance descriptors (CovDs) into a rich descriptor, which in essence benefit from the second-order statistics along the coding pipeline. The difficulty in aggregating CovDs arises from the fact that CovDs lie on the Riemannian manifold of symmetric positive definite (SPD) matrices. Therefore, the aggregating scheme must take advantage of metrics and the geometry of the SPD manifolds. In our proposal, we make use of the Stein divergence and Nyström method to embed the SPD manifold into a Hilbert space. We compare our proposal, dubbed CovLets, against state-of-the-art methods on several image and video classification problems including facial expression recognition and action recognition.