A Two-Stream CNN Framework for American Sign Language Recognition Based on Multimodal Data Fusion

Qing Gao; Uchenna Emeoha Ogenyi; Jinguo Liu; Zhaojie Ju; Honghai Liu

doi:10.1007/978-3-030-29933-0_9

A Two-Stream CNN Framework for American Sign Language Recognition Based on Multimodal Data Fusion

Qing Gao
, Uchenna Emeoha Ogenyi
, Jinguo Liu^*
, Zhaojie Ju
, Honghai Liu

^*Corresponding author for this work

CAS - Shenyang Institute of Automation
Chinese Academy of Sciences
University of Chinese Academy of Sciences
University of Portsmouth

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

21 Scopus citations

Abstract

At present, vision-based hand gesture recognition is very important in human-robot interaction (HRI). This non-contact method enables natural and friendly interaction between people and robots. Aiming at this technology, a two-stream CNN framework (2S-CNN) is proposed to recognize the American sign language (ASL) hand gestures based on multimodal (RGB and depth) data fusion. Firstly, the hand gesture data is enhanced to remove the influence of background and noise. Secondly, hand gesture RGB and depth features are extracted for hand gesture recognition using CNNs on two streams, respectively. Finally, a fusion layer is designed for fusing the recognition results of the two streams. This method utilizes multimodal data to increase the recognition accuracy of the ASL hand gestures. The experiments prove that the recognition accuracy of 2S-CNN can reach 92.08 $$\%$$ on ASL fingerspelling database and is higher than that of baseline methods.

Original language	English
Title of host publication	Advances in Computational Intelligence Systems - Contributions Presented at the 19th UK Workshop on Computational Intelligence, 2019
Editors	Zhaojie Ju, Dalin Zhou, Alexander Gegov, Longzhi Yang, Chenguang Yang
Publisher	Springer Verlag
Pages	107-118
Number of pages	12
ISBN (Print)	9783030299323
DOIs	https://doi.org/10.1007/978-3-030-29933-0_9
State	Published - 2020
Externally published	Yes
Event	19th Annual UK Workshop on Computational Intelligence, UKCI 2019 - Portsmouth, United Kingdom Duration: 4 Sep 2019 → 6 Sep 2019

Publication series

Name	Advances in Intelligent Systems and Computing
Volume	1043
ISSN (Print)	2194-5357
ISSN (Electronic)	2194-5365

Conference

Conference	19th Annual UK Workshop on Computational Intelligence, UKCI 2019
Country/Territory	United Kingdom
City	Portsmouth
Period	4/09/19 → 6/09/19

Keywords

CNN
Hand gesture recognition
Multimodal data fusion

Access to Document

10.1007/978-3-030-29933-0_9

Cite this

Gao, Q., Ogenyi, U. E., Liu, J., Ju, Z., & Liu, H. (2020). A Two-Stream CNN Framework for American Sign Language Recognition Based on Multimodal Data Fusion. In Z. Ju, D. Zhou, A. Gegov, L. Yang, & C. Yang (Eds.), Advances in Computational Intelligence Systems - Contributions Presented at the 19th UK Workshop on Computational Intelligence, 2019 (pp. 107-118). (Advances in Intelligent Systems and Computing; Vol. 1043). Springer Verlag. https://doi.org/10.1007/978-3-030-29933-0_9

Gao, Qing ; Ogenyi, Uchenna Emeoha ; Liu, Jinguo et al. / A Two-Stream CNN Framework for American Sign Language Recognition Based on Multimodal Data Fusion. Advances in Computational Intelligence Systems - Contributions Presented at the 19th UK Workshop on Computational Intelligence, 2019. editor / Zhaojie Ju ; Dalin Zhou ; Alexander Gegov ; Longzhi Yang ; Chenguang Yang. Springer Verlag, 2020. pp. 107-118 (Advances in Intelligent Systems and Computing).

@inproceedings{2ae94c9d11484c1881ee94237ca3bed6,

title = "A Two-Stream CNN Framework for American Sign Language Recognition Based on Multimodal Data Fusion",

abstract = "At present, vision-based hand gesture recognition is very important in human-robot interaction (HRI). This non-contact method enables natural and friendly interaction between people and robots. Aiming at this technology, a two-stream CNN framework (2S-CNN) is proposed to recognize the American sign language (ASL) hand gestures based on multimodal (RGB and depth) data fusion. Firstly, the hand gesture data is enhanced to remove the influence of background and noise. Secondly, hand gesture RGB and depth features are extracted for hand gesture recognition using CNNs on two streams, respectively. Finally, a fusion layer is designed for fusing the recognition results of the two streams. This method utilizes multimodal data to increase the recognition accuracy of the ASL hand gestures. The experiments prove that the recognition accuracy of 2S-CNN can reach 92.08 \$\$\textbackslash{}\%\$\$ on ASL fingerspelling database and is higher than that of baseline methods.",

keywords = "CNN, Hand gesture recognition, Multimodal data fusion",

author = "Qing Gao and Ogenyi, \{Uchenna Emeoha\} and Jinguo Liu and Zhaojie Ju and Honghai Liu",

note = "Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; 19th Annual UK Workshop on Computational Intelligence, UKCI 2019 ; Conference date: 04-09-2019 Through 06-09-2019",

year = "2020",

doi = "10.1007/978-3-030-29933-0\_9",

language = "英语",

isbn = "9783030299323",

series = "Advances in Intelligent Systems and Computing",

publisher = "Springer Verlag",

pages = "107--118",

editor = "Zhaojie Ju and Dalin Zhou and Alexander Gegov and Longzhi Yang and Chenguang Yang",

booktitle = "Advances in Computational Intelligence Systems - Contributions Presented at the 19th UK Workshop on Computational Intelligence, 2019",

address = "德国",

}

Gao, Q, Ogenyi, UE, Liu, J, Ju, Z & Liu, H 2020, A Two-Stream CNN Framework for American Sign Language Recognition Based on Multimodal Data Fusion. in Z Ju, D Zhou, A Gegov, L Yang & C Yang (eds), Advances in Computational Intelligence Systems - Contributions Presented at the 19th UK Workshop on Computational Intelligence, 2019. Advances in Intelligent Systems and Computing, vol. 1043, Springer Verlag, pp. 107-118, 19th Annual UK Workshop on Computational Intelligence, UKCI 2019, Portsmouth, United Kingdom, 4/09/19. https://doi.org/10.1007/978-3-030-29933-0_9

A Two-Stream CNN Framework for American Sign Language Recognition Based on Multimodal Data Fusion. / Gao, Qing; Ogenyi, Uchenna Emeoha; Liu, Jinguo et al.
Advances in Computational Intelligence Systems - Contributions Presented at the 19th UK Workshop on Computational Intelligence, 2019. ed. / Zhaojie Ju; Dalin Zhou; Alexander Gegov; Longzhi Yang; Chenguang Yang. Springer Verlag, 2020. p. 107-118 (Advances in Intelligent Systems and Computing; Vol. 1043).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A Two-Stream CNN Framework for American Sign Language Recognition Based on Multimodal Data Fusion

AU - Gao, Qing

AU - Ogenyi, Uchenna Emeoha

AU - Liu, Jinguo

AU - Ju, Zhaojie

AU - Liu, Honghai

PY - 2020

Y1 - 2020

N2 - At present, vision-based hand gesture recognition is very important in human-robot interaction (HRI). This non-contact method enables natural and friendly interaction between people and robots. Aiming at this technology, a two-stream CNN framework (2S-CNN) is proposed to recognize the American sign language (ASL) hand gestures based on multimodal (RGB and depth) data fusion. Firstly, the hand gesture data is enhanced to remove the influence of background and noise. Secondly, hand gesture RGB and depth features are extracted for hand gesture recognition using CNNs on two streams, respectively. Finally, a fusion layer is designed for fusing the recognition results of the two streams. This method utilizes multimodal data to increase the recognition accuracy of the ASL hand gestures. The experiments prove that the recognition accuracy of 2S-CNN can reach 92.08 $$\%$$ on ASL fingerspelling database and is higher than that of baseline methods.

AB - At present, vision-based hand gesture recognition is very important in human-robot interaction (HRI). This non-contact method enables natural and friendly interaction between people and robots. Aiming at this technology, a two-stream CNN framework (2S-CNN) is proposed to recognize the American sign language (ASL) hand gestures based on multimodal (RGB and depth) data fusion. Firstly, the hand gesture data is enhanced to remove the influence of background and noise. Secondly, hand gesture RGB and depth features are extracted for hand gesture recognition using CNNs on two streams, respectively. Finally, a fusion layer is designed for fusing the recognition results of the two streams. This method utilizes multimodal data to increase the recognition accuracy of the ASL hand gestures. The experiments prove that the recognition accuracy of 2S-CNN can reach 92.08 $$\%$$ on ASL fingerspelling database and is higher than that of baseline methods.

KW - CNN

KW - Hand gesture recognition

KW - Multimodal data fusion

UR - https://www.scopus.com/pages/publications/85072869151

U2 - 10.1007/978-3-030-29933-0_9

DO - 10.1007/978-3-030-29933-0_9

M3 - 会议稿件

AN - SCOPUS:85072869151

SN - 9783030299323

T3 - Advances in Intelligent Systems and Computing

SP - 107

EP - 118

BT - Advances in Computational Intelligence Systems - Contributions Presented at the 19th UK Workshop on Computational Intelligence, 2019

A2 - Ju, Zhaojie

A2 - Zhou, Dalin

A2 - Gegov, Alexander

A2 - Yang, Longzhi

A2 - Yang, Chenguang

PB - Springer Verlag

T2 - 19th Annual UK Workshop on Computational Intelligence, UKCI 2019

Y2 - 4 September 2019 through 6 September 2019

ER -

Gao Q, Ogenyi UE, Liu J, Ju Z, Liu H. A Two-Stream CNN Framework for American Sign Language Recognition Based on Multimodal Data Fusion. In Ju Z, Zhou D, Gegov A, Yang L, Yang C, editors, Advances in Computational Intelligence Systems - Contributions Presented at the 19th UK Workshop on Computational Intelligence, 2019. Springer Verlag. 2020. p. 107-118. (Advances in Intelligent Systems and Computing). doi: 10.1007/978-3-030-29933-0_9

A Two-Stream CNN Framework for American Sign Language Recognition Based on Multimodal Data Fusion

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this