Average reward reinforcement learning for semi-markov decision processes

Jiayuan Yang; Yanjie Li; Haoyao Chen; Jiangang Li

doi:10.1007/978-3-319-70087-8_79

Average reward reinforcement learning for semi-markov decision processes

Jiayuan Yang
, Yanjie Li^*
, Haoyao Chen
, Jiangang Li

^*Corresponding author for this work

Harbin Institute of Technology Shenzhen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

In this paper, we study new reinforcement learning (RL) algorithms for Semi-Markov decision processes (SMDPs) with an average reward criterion. Based on the discrete-time type Bellman optimality equation, we use incremental value iteration (IVI), stochastic shortest path (SSP) value iteration and bisection algorithms to derive novel RL algorithms in a straightforward way. These algorithms use IVI, SSP and dichotomy to directly estimate the optimal average reward to solve the instability of average reward RL, respectively. Furthermore, a simulation experiment is used to compare the convergence among these algorithms.

Original language	English
Title of host publication	Neural Information Processing - 24th International Conference, ICONIP 2017, Proceedings
Editors	Yuanqing Li, Derong Liu, Shengli Xie, El-Sayed M. El-Alfy, Dongbin Zhao
Publisher	Springer Verlag
Pages	768-777
Number of pages	10
ISBN (Print)	9783319700861
DOIs	https://doi.org/10.1007/978-3-319-70087-8_79
State	Published - 2017
Externally published	Yes
Event	24th International Conference on Neural Information Processing, ICONIP 2017 - Guangzhou, China Duration: 14 Nov 2017 → 18 Nov 2017

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	10634 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	24th International Conference on Neural Information Processing, ICONIP 2017
Country/Territory	China
City	Guangzhou
Period	14/11/17 → 18/11/17

Keywords

Incremental value iteration
SMDPs
Stochastic shortest path

Access to Document

10.1007/978-3-319-70087-8_79

Cite this

Yang, J., Li, Y., Chen, H., & Li, J. (2017). Average reward reinforcement learning for semi-markov decision processes. In Y. Li, D. Liu, S. Xie, E.-S. M. El-Alfy, & D. Zhao (Eds.), Neural Information Processing - 24th International Conference, ICONIP 2017, Proceedings (pp. 768-777). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10634 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-70087-8_79

Yang, Jiayuan ; Li, Yanjie ; Chen, Haoyao et al. / Average reward reinforcement learning for semi-markov decision processes. Neural Information Processing - 24th International Conference, ICONIP 2017, Proceedings. editor / Yuanqing Li ; Derong Liu ; Shengli Xie ; El-Sayed M. El-Alfy ; Dongbin Zhao. Springer Verlag, 2017. pp. 768-777 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{4f8f7873e493472686e77394240db1a6,

title = "Average reward reinforcement learning for semi-markov decision processes",

abstract = "In this paper, we study new reinforcement learning (RL) algorithms for Semi-Markov decision processes (SMDPs) with an average reward criterion. Based on the discrete-time type Bellman optimality equation, we use incremental value iteration (IVI), stochastic shortest path (SSP) value iteration and bisection algorithms to derive novel RL algorithms in a straightforward way. These algorithms use IVI, SSP and dichotomy to directly estimate the optimal average reward to solve the instability of average reward RL, respectively. Furthermore, a simulation experiment is used to compare the convergence among these algorithms.",

keywords = "Incremental value iteration, SMDPs, Stochastic shortest path",

author = "Jiayuan Yang and Yanjie Li and Haoyao Chen and Jiangang Li",

note = "Publisher Copyright: {\textcopyright} Springer International Publishing AG 2017.; 24th International Conference on Neural Information Processing, ICONIP 2017 ; Conference date: 14-11-2017 Through 18-11-2017",

year = "2017",

doi = "10.1007/978-3-319-70087-8\_79",

language = "英语",

isbn = "9783319700861",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "768--777",

editor = "Yuanqing Li and Derong Liu and Shengli Xie and El-Alfy, \{El-Sayed M.\} and Dongbin Zhao",

booktitle = "Neural Information Processing - 24th International Conference, ICONIP 2017, Proceedings",

address = "德国",

}

Yang, J, Li, Y , Chen, H & Li, J 2017, Average reward reinforcement learning for semi-markov decision processes. in Y Li, D Liu, S Xie, E-SM El-Alfy & D Zhao (eds), Neural Information Processing - 24th International Conference, ICONIP 2017, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10634 LNCS, Springer Verlag, pp. 768-777, 24th International Conference on Neural Information Processing, ICONIP 2017, Guangzhou, China, 14/11/17. https://doi.org/10.1007/978-3-319-70087-8_79

Average reward reinforcement learning for semi-markov decision processes. / Yang, Jiayuan; Li, Yanjie ; Chen, Haoyao et al.
Neural Information Processing - 24th International Conference, ICONIP 2017, Proceedings. ed. / Yuanqing Li; Derong Liu; Shengli Xie; El-Sayed M. El-Alfy; Dongbin Zhao. Springer Verlag, 2017. p. 768-777 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10634 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Average reward reinforcement learning for semi-markov decision processes

AU - Yang, Jiayuan

AU - Li, Yanjie

AU - Chen, Haoyao

AU - Li, Jiangang

N1 - Publisher Copyright: © Springer International Publishing AG 2017.

PY - 2017

Y1 - 2017

N2 - In this paper, we study new reinforcement learning (RL) algorithms for Semi-Markov decision processes (SMDPs) with an average reward criterion. Based on the discrete-time type Bellman optimality equation, we use incremental value iteration (IVI), stochastic shortest path (SSP) value iteration and bisection algorithms to derive novel RL algorithms in a straightforward way. These algorithms use IVI, SSP and dichotomy to directly estimate the optimal average reward to solve the instability of average reward RL, respectively. Furthermore, a simulation experiment is used to compare the convergence among these algorithms.

AB - In this paper, we study new reinforcement learning (RL) algorithms for Semi-Markov decision processes (SMDPs) with an average reward criterion. Based on the discrete-time type Bellman optimality equation, we use incremental value iteration (IVI), stochastic shortest path (SSP) value iteration and bisection algorithms to derive novel RL algorithms in a straightforward way. These algorithms use IVI, SSP and dichotomy to directly estimate the optimal average reward to solve the instability of average reward RL, respectively. Furthermore, a simulation experiment is used to compare the convergence among these algorithms.

KW - Incremental value iteration

KW - SMDPs

KW - Stochastic shortest path

UR - https://www.scopus.com/pages/publications/85035143316

U2 - 10.1007/978-3-319-70087-8_79

DO - 10.1007/978-3-319-70087-8_79

M3 - 会议稿件

AN - SCOPUS:85035143316

SN - 9783319700861

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 768

EP - 777

BT - Neural Information Processing - 24th International Conference, ICONIP 2017, Proceedings

A2 - Li, Yuanqing

A2 - Liu, Derong

A2 - Xie, Shengli

A2 - El-Alfy, El-Sayed M.

A2 - Zhao, Dongbin

PB - Springer Verlag

T2 - 24th International Conference on Neural Information Processing, ICONIP 2017

Y2 - 14 November 2017 through 18 November 2017

ER -

Yang J, Li Y , Chen H , Li J. Average reward reinforcement learning for semi-markov decision processes. In Li Y, Liu D, Xie S, El-Alfy ESM, Zhao D, editors, Neural Information Processing - 24th International Conference, ICONIP 2017, Proceedings. Springer Verlag. 2017. p. 768-777. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-70087-8_79

Average reward reinforcement learning for semi-markov decision processes

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this