PSRMER: Proactive Services Recommendation Driven-by Multimodal Emotion Recognition

In the past three years, global COVID-19 pandemic not only impacted people's physical health but also significantly affected their mental health, which resulting in rapid increase of psychological problems. Emotions are a common manifestation of psychological changes, and some services (such as music, video, or psychological counseling services) can help users to adjust their emotions in a timely manner, thus to avoid bringing extreme events (e.g., running away from home or committing suicide). Therefore, how to perceive users' real-time emotions and then recommend the most appropriate services to users has become a challenge. To address this issue, this work proposes an approach for proactive services recommendation driven-by multimodal emotion recognition (named as PSRMER). Specifically, PSRMER first actively identifies a user's emotion with a multimodal emotion recognition model based on BiGRU and Transformer; Then, considering the user's emotion and preferences, PSRMER selects the optimal services based on an index-graph linking different emotions and various services; Finally, PSRMER proactively recommends the selected optimal service to the user. Extensive experiments have been conducted and the effectiveness of our proposed method have been proved. Moreover, the proposed method can also be used in smart education, smart transportation, smart elderly care and other modern industry fields.