Exploration via Distributional Reinforcement Learning with Epistemic and Aleatoric Uncertainty Estimation

Qi Liu; Yanjie Li; Yuecheng Liu; Meiling Chen; Shaohua Lv; Yunhong Xu

doi:10.1109/CASE49439.2021.9551544

Exploration via Distributional Reinforcement Learning with Epistemic and Aleatoric Uncertainty Estimation

Qi Liu
, Yanjie Li
, Yuecheng Liu
, Meiling Chen
, Shaohua Lv
, Yunhong Xu

Harbin Institute of Technology Shenzhen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

5 Scopus citations

Abstract

The problem of exploration remains one of the major challenges in deep reinforcement learning (RL). This paper proposes an approach to improve the exploration efficiency for distributional RL. First, this paper proposes a novel method to estimate the epistemic and aleatoric uncertainty for distributional RL using deep ensembles, which is inspired by Bayesian Deep Learning. Second, This paper presents a method to improve the exploration efficiency for deep distributional RL by using estimated epistemic uncertainty. Experimental results show that the proposed approach outperforms the baseline in Atari games.

Original language	English
Title of host publication	2021 IEEE 17th International Conference on Automation Science and Engineering, CASE 2021
Publisher	IEEE Computer Society
Pages	2256-2261
Number of pages	6
ISBN (Electronic)	9781665418737
DOIs	https://doi.org/10.1109/CASE49439.2021.9551544
State	Published - 23 Aug 2021
Externally published	Yes
Event	17th IEEE International Conference on Automation Science and Engineering, CASE 2021 - Lyon, France Duration: 23 Aug 2021 → 27 Aug 2021

Publication series

Name	IEEE International Conference on Automation Science and Engineering
Volume	2021-August
ISSN (Print)	2161-8070
ISSN (Electronic)	2161-8089

Conference

Conference	17th IEEE International Conference on Automation Science and Engineering, CASE 2021
Country/Territory	France
City	Lyon
Period	23/08/21 → 27/08/21

Access to Document

10.1109/CASE49439.2021.9551544

Cite this

Liu, Q., Li, Y., Liu, Y., Chen, M., Lv, S., & Xu, Y. (2021). Exploration via Distributional Reinforcement Learning with Epistemic and Aleatoric Uncertainty Estimation. In 2021 IEEE 17th International Conference on Automation Science and Engineering, CASE 2021 (pp. 2256-2261). (IEEE International Conference on Automation Science and Engineering; Vol. 2021-August). IEEE Computer Society. https://doi.org/10.1109/CASE49439.2021.9551544

@inproceedings{b3d6284ccb7f4302ba0bc41f3ca096fa,

title = "Exploration via Distributional Reinforcement Learning with Epistemic and Aleatoric Uncertainty Estimation",

abstract = "The problem of exploration remains one of the major challenges in deep reinforcement learning (RL). This paper proposes an approach to improve the exploration efficiency for distributional RL. First, this paper proposes a novel method to estimate the epistemic and aleatoric uncertainty for distributional RL using deep ensembles, which is inspired by Bayesian Deep Learning. Second, This paper presents a method to improve the exploration efficiency for deep distributional RL by using estimated epistemic uncertainty. Experimental results show that the proposed approach outperforms the baseline in Atari games.",

author = "Qi Liu and Yanjie Li and Yuecheng Liu and Meiling Chen and Shaohua Lv and Yunhong Xu",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE.; 17th IEEE International Conference on Automation Science and Engineering, CASE 2021 ; Conference date: 23-08-2021 Through 27-08-2021",

year = "2021",

month = aug,

day = "23",

doi = "10.1109/CASE49439.2021.9551544",

language = "英语",

series = "IEEE International Conference on Automation Science and Engineering",

publisher = "IEEE Computer Society",

pages = "2256--2261",

booktitle = "2021 IEEE 17th International Conference on Automation Science and Engineering, CASE 2021",

address = "美国",

}

Liu, Q, Li, Y, Liu, Y, Chen, M, Lv, S & Xu, Y 2021, Exploration via Distributional Reinforcement Learning with Epistemic and Aleatoric Uncertainty Estimation. in 2021 IEEE 17th International Conference on Automation Science and Engineering, CASE 2021. IEEE International Conference on Automation Science and Engineering, vol. 2021-August, IEEE Computer Society, pp. 2256-2261, 17th IEEE International Conference on Automation Science and Engineering, CASE 2021, Lyon, France, 23/08/21. https://doi.org/10.1109/CASE49439.2021.9551544

Exploration via Distributional Reinforcement Learning with Epistemic and Aleatoric Uncertainty Estimation. / Liu, Qi; Li, Yanjie; Liu, Yuecheng et al.
2021 IEEE 17th International Conference on Automation Science and Engineering, CASE 2021. IEEE Computer Society, 2021. p. 2256-2261 (IEEE International Conference on Automation Science and Engineering; Vol. 2021-August).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Exploration via Distributional Reinforcement Learning with Epistemic and Aleatoric Uncertainty Estimation

AU - Liu, Qi

AU - Li, Yanjie

AU - Liu, Yuecheng

AU - Chen, Meiling

AU - Lv, Shaohua

AU - Xu, Yunhong

PY - 2021/8/23

Y1 - 2021/8/23

N2 - The problem of exploration remains one of the major challenges in deep reinforcement learning (RL). This paper proposes an approach to improve the exploration efficiency for distributional RL. First, this paper proposes a novel method to estimate the epistemic and aleatoric uncertainty for distributional RL using deep ensembles, which is inspired by Bayesian Deep Learning. Second, This paper presents a method to improve the exploration efficiency for deep distributional RL by using estimated epistemic uncertainty. Experimental results show that the proposed approach outperforms the baseline in Atari games.

AB - The problem of exploration remains one of the major challenges in deep reinforcement learning (RL). This paper proposes an approach to improve the exploration efficiency for distributional RL. First, this paper proposes a novel method to estimate the epistemic and aleatoric uncertainty for distributional RL using deep ensembles, which is inspired by Bayesian Deep Learning. Second, This paper presents a method to improve the exploration efficiency for deep distributional RL by using estimated epistemic uncertainty. Experimental results show that the proposed approach outperforms the baseline in Atari games.

UR - https://www.scopus.com/pages/publications/85116966569

U2 - 10.1109/CASE49439.2021.9551544

DO - 10.1109/CASE49439.2021.9551544

M3 - 会议稿件

AN - SCOPUS:85116966569

T3 - IEEE International Conference on Automation Science and Engineering

SP - 2256

EP - 2261

BT - 2021 IEEE 17th International Conference on Automation Science and Engineering, CASE 2021

PB - IEEE Computer Society

T2 - 17th IEEE International Conference on Automation Science and Engineering, CASE 2021

Y2 - 23 August 2021 through 27 August 2021

ER -

Liu Q, Li Y, Liu Y, Chen M, Lv S, Xu Y. Exploration via Distributional Reinforcement Learning with Epistemic and Aleatoric Uncertainty Estimation. In 2021 IEEE 17th International Conference on Automation Science and Engineering, CASE 2021. IEEE Computer Society. 2021. p. 2256-2261. (IEEE International Conference on Automation Science and Engineering). doi: 10.1109/CASE49439.2021.9551544

Exploration via Distributional Reinforcement Learning with Epistemic and Aleatoric Uncertainty Estimation

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this