Facial landmarks detection under occlusions via extended restricted boltzmann machine

Facial landmarks encode critical information about face, which plays an important role in human communications. Accurate detecting and tracking facial landmarks have great potential value in intelligent user interfaces such as human-computer interactions. However, for face images with sever occlusions which may happen in real life such as hand occlusion, gesture occlusion and etc, detecting the facial landmarks is still a challenging problem. In this paper, we present a robust facial landmark detection method for image with occlusions based on Restricted Boltzmann Machine (RBM). We first present a face shape prior model which is constructed based on RBM to model the spatial shape patterns of the face. The detection process is accomplished by combining the prior shape model with the image measurements of facial landmarks. The low accuracy image measurements can be refined by the shape information embedded in the prior model. For the landmarks with severe occlusions, we firstly evaluate and determine the facial landmark occlusions, and replace their image measurements. The new image measurements are then fed into the prior model as evidence to predict the true locations. Evaluation on 3 databases demonstrates that the proposed method can detect facial landmarks accurately under severe occlusion, and achieved significant improvement over the current state of the art methods.