On-line policy gradient estimation with multi-step sampling

Yan Jie Li; Fang Cao; Xi Ren Cao

doi:10.1007/s10626-009-0078-3

On-line policy gradient estimation with multi-step sampling

Yan Jie Li^*
, Fang Cao
, Xi Ren Cao

^*Corresponding author for this work

Hong Kong University of Science and Technology
Harbin Institute of Technology Shenzhen
Beijing Jiaotong University

Research output: Contribution to journal › Article › peer-review

5 Scopus citations

Abstract

In this note, we discuss the problem of the sample-path-based (on-line) performance gradient estimation for Markov systems. The existing on-line performance gradient estimation algorithms generally require a standard importance sampling assumption. When the assumption does not hold, these algorithms may lead to poor estimates for the gradients. We show that this assumption can be relaxed and propose algorithms with multi-step sampling for performance gradient estimates; these algorithms do not require the standard assumption. Simulation examples are given to illustrate the accuracy of the estimates.

Original language	English
Pages (from-to)	3-17
Number of pages	15
Journal	Discrete Event Dynamic Systems: Theory and Applications
Volume	20
Issue number	1
DOIs	https://doi.org/10.1007/s10626-009-0078-3
State	Published - Mar 2010
Externally published	Yes

Keywords

Markov reward processes
On-line estimation
Performance potentials
Policy gradient

Access to Document

10.1007/s10626-009-0078-3

Cite this

@article{398fe07eada84172ab92b06fd1bbbb81,

title = "On-line policy gradient estimation with multi-step sampling",

abstract = "In this note, we discuss the problem of the sample-path-based (on-line) performance gradient estimation for Markov systems. The existing on-line performance gradient estimation algorithms generally require a standard importance sampling assumption. When the assumption does not hold, these algorithms may lead to poor estimates for the gradients. We show that this assumption can be relaxed and propose algorithms with multi-step sampling for performance gradient estimates; these algorithms do not require the standard assumption. Simulation examples are given to illustrate the accuracy of the estimates.",

keywords = "Markov reward processes, On-line estimation, Performance potentials, Policy gradient",

author = "Li, \{Yan Jie\} and Fang Cao and Cao, \{Xi Ren\}",

year = "2010",

month = mar,

doi = "10.1007/s10626-009-0078-3",

language = "英语",

volume = "20",

pages = "3--17",

journal = "Discrete Event Dynamic Systems: Theory and Applications",

issn = "0924-6703",

publisher = "Springer Netherlands",

number = "1",

}

TY - JOUR

T1 - On-line policy gradient estimation with multi-step sampling

AU - Li, Yan Jie

AU - Cao, Fang

AU - Cao, Xi Ren

PY - 2010/3

Y1 - 2010/3

N2 - In this note, we discuss the problem of the sample-path-based (on-line) performance gradient estimation for Markov systems. The existing on-line performance gradient estimation algorithms generally require a standard importance sampling assumption. When the assumption does not hold, these algorithms may lead to poor estimates for the gradients. We show that this assumption can be relaxed and propose algorithms with multi-step sampling for performance gradient estimates; these algorithms do not require the standard assumption. Simulation examples are given to illustrate the accuracy of the estimates.

AB - In this note, we discuss the problem of the sample-path-based (on-line) performance gradient estimation for Markov systems. The existing on-line performance gradient estimation algorithms generally require a standard importance sampling assumption. When the assumption does not hold, these algorithms may lead to poor estimates for the gradients. We show that this assumption can be relaxed and propose algorithms with multi-step sampling for performance gradient estimates; these algorithms do not require the standard assumption. Simulation examples are given to illustrate the accuracy of the estimates.

KW - Markov reward processes

KW - On-line estimation

KW - Performance potentials

KW - Policy gradient

UR - https://www.scopus.com/pages/publications/76649119454

U2 - 10.1007/s10626-009-0078-3

DO - 10.1007/s10626-009-0078-3

M3 - 文章

AN - SCOPUS:76649119454

SN - 0924-6703

VL - 20

SP - 3

EP - 17

JO - Discrete Event Dynamic Systems: Theory and Applications

JF - Discrete Event Dynamic Systems: Theory and Applications

IS - 1

ER -

On-line policy gradient estimation with multi-step sampling

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this