Online Least-squares Policy Iteration For Reinforcement Learning Control.pdf

Online Least-Squares Policy Iteration for Reinforcement Learning Control

This technical report by Delft University of Technology's Delft Center for Systems and Control presents online least-squares policy iteration (LSPI) for reinforcement learning (RL) control.

Problem:

The paper tackles the core RL problem: finding an optimal policy that maximizes cumulative rewards in a stochastic, nonlinear environment through interaction. Traditional RL algorithms use value functions (like Q-functions) to represent expected future returns and policy iteration (PI) methods iteratively improve policies by evaluating these values.

Existing Approaches:

Offline LSPI: Existing LSPI methods evaluate policies thoroughly before updating them, requiring many samples for accurate Q-function estimation. This is suitable for offline training but not real-time applications.
Gradient-based Evaluation: Many online PI algorithms rely on gradient-based policy evaluation, which can be less efficient than least-squares methods.

Proposed Solution: Online LSPI with LSTD-Q

The authors introduce online LSPI, a novel method that:

1. Optimistic Updates: Improves policies after only a few state transitions, using an incomplete evaluation of the current policy. This allows for real-time learning.
2. Least-Squares Temporal Difference (LSTD-Q): Employs LSTD-Q for Q-function estimation, known for its sample efficiency and relaxed convergence requirements.

Key Differences:

Online LSPI collects its own samples through interaction with the environment.
It performs "optimistic" policy updates before a complete evaluation of the current policy is possible.

Experimental Evaluation:

The paper demonstrates online LSPI's effectiveness through:

Theoretical Analysis: Deriving convergence guarantees for online LSPI.
Real-Time Control: Successfully applying online LSPI to swing up an underactuated inverted pendulum in real time.
* Comparison: Comparing online LSPI with offline LSPI and a competing least-squares method (LSPE-Q), showcasing its advantages in terms of learning speed and performance.

Conclusion:

The proposed online LSPI algorithm offers a promising approach for real-time reinforcement learning control, combining efficiency and accuracy through the use of least-squares techniques.