一种基于多任务学习的多模态情感识别方法

Zijie Lin; Yunfei Long; Jiachen Du; Ruifeng Xu

doi:10.13209/j.0479-8023.2020.085

一种基于多任务学习的多模态情感识别方法

Translated title of the contribution: A Multi-modal Sentiment Recognition Method Based on Multi-task Learning

Zijie Lin
, Yunfei Long
, Jiachen Du
, Ruifeng Xu^*

^*Corresponding author for this work

Harbin Institute of Technology
University of Essex

Research output: Contribution to journal › Article › peer-review

7 Scopus citations

Abstract

In order to learn more emotionally inclined video and speech representations through auxiliary tasks, and improve the effect of multi-modal fusion, this paper proposes a multi-modal sentiment recognition method based on multi-task learning. A multimodal sharing layer is used to learn the sentiment information of the visual and acoustic modes. The experiment on MOSI and MOSEI data sets shows that adding two auxiliary single-modal sentiment recognition tasks can learn more effective single-modal sentiment representations, and improve the accuracy of sentiment recognition by 0.8% and 2.5% respectively.

Translated title of the contribution	A Multi-modal Sentiment Recognition Method Based on Multi-task Learning
Original language	Chinese (Traditional)
Pages (from-to)	7-15
Number of pages	9
Journal	Beijing Daxue Xuebao (Ziran Kexue Ban)/Acta Scientiarum Naturalium Universitatis Pekinensis
Volume	57
Issue number	1
DOIs	https://doi.org/10.13209/j.0479-8023.2020.085
State	Published - 20 Jan 2021
Externally published	Yes

Access to Document

10.13209/j.0479-8023.2020.085

Cite this

@article{0beebadd91314a3e8926e2c0a28068df,

title = "一种基于多任务学习的多模态情感识别方法",

abstract = "In order to learn more emotionally inclined video and speech representations through auxiliary tasks, and improve the effect of multi-modal fusion, this paper proposes a multi-modal sentiment recognition method based on multi-task learning. A multimodal sharing layer is used to learn the sentiment information of the visual and acoustic modes. The experiment on MOSI and MOSEI data sets shows that adding two auxiliary single-modal sentiment recognition tasks can learn more effective single-modal sentiment representations, and improve the accuracy of sentiment recognition by 0.8\% and 2.5\% respectively.",

keywords = "Multi-modal fusion, Multi-modal information, Multi-task learning, Sentiment recognition",

author = "Zijie Lin and Yunfei Long and Jiachen Du and Ruifeng Xu",

note = "Publisher Copyright: {\textcopyright} 2021 Peking University.",

year = "2021",

month = jan,

day = "20",

doi = "10.13209/j.0479-8023.2020.085",

language = "繁体中文",

volume = "57",

pages = "7--15",

journal = "Beijing Daxue Xuebao (Ziran Kexue Ban)/Acta Scientiarum Naturalium Universitatis Pekinensis",

issn = "0479-8023",

publisher = "Peking University",

number = "1",

}

TY - JOUR

T1 - 一种基于多任务学习的多模态情感识别方法

AU - Lin, Zijie

AU - Long, Yunfei

AU - Du, Jiachen

AU - Xu, Ruifeng

PY - 2021/1/20

Y1 - 2021/1/20

N2 - In order to learn more emotionally inclined video and speech representations through auxiliary tasks, and improve the effect of multi-modal fusion, this paper proposes a multi-modal sentiment recognition method based on multi-task learning. A multimodal sharing layer is used to learn the sentiment information of the visual and acoustic modes. The experiment on MOSI and MOSEI data sets shows that adding two auxiliary single-modal sentiment recognition tasks can learn more effective single-modal sentiment representations, and improve the accuracy of sentiment recognition by 0.8% and 2.5% respectively.

AB - In order to learn more emotionally inclined video and speech representations through auxiliary tasks, and improve the effect of multi-modal fusion, this paper proposes a multi-modal sentiment recognition method based on multi-task learning. A multimodal sharing layer is used to learn the sentiment information of the visual and acoustic modes. The experiment on MOSI and MOSEI data sets shows that adding two auxiliary single-modal sentiment recognition tasks can learn more effective single-modal sentiment representations, and improve the accuracy of sentiment recognition by 0.8% and 2.5% respectively.

KW - Multi-modal fusion

KW - Multi-modal information

KW - Multi-task learning

KW - Sentiment recognition

UR - https://www.scopus.com/pages/publications/85101340819

U2 - 10.13209/j.0479-8023.2020.085

DO - 10.13209/j.0479-8023.2020.085

M3 - 文章

AN - SCOPUS:85101340819

SN - 0479-8023

VL - 57

SP - 7

EP - 15

JO - Beijing Daxue Xuebao (Ziran Kexue Ban)/Acta Scientiarum Naturalium Universitatis Pekinensis

JF - Beijing Daxue Xuebao (Ziran Kexue Ban)/Acta Scientiarum Naturalium Universitatis Pekinensis

IS - 1

ER -

一种基于多任务学习的多模态情感识别方法

Abstract

Access to Document

Other files and links

Fingerprint

Cite this