[David Silver] 7. Policy Gradient: REINFORCE, Actor-Critic, NPG

이 글은 필자가 David Silver의 Reinforcement Learning 강좌를 듣고 정리한 글입니다.

Last lecture, we approximated the value or action-value function using parameters $\theta$. Policy was generated directly from the value function. In this lecture, we will directly parameterise the policy as stochastic

$\pi_\theta(s,a) = \mathbb{P} [a|s, \theta]$

This taxonomy explains Value-based and Policy-based RL well.

Value-Based and Policy-based RL. David Silver RL lecture 7.

• Value-based RL: Learnt value function and implicit (deterministic) policy (e.g. $\epsilon$-greedy)
  • Ex. DQN
• Policy-based RL: No value function but learnt (stochastic)policy
  • Ex. REINFORCE
  • Policy can be softmax policy / Gaussian policy
  •  Advantage
    1. Better convergence properties
    2. Effective in high-dimensional or continuous action spaces
    3. Can learn stochastic policies
  • Disadvantage
    1. Typically converge to a local rather than global optimum
    2. Evaluating a policy is typically inefficient and high variance
• Value, Policy-based RL: Learnt value function and Learnt policy
  • Ex. Actor-Critic

🥏 Policy Gradient with Policy Objective Functions

• Goal: Given policy $\pi_\theta (s,a)$ with parameters $\theta$, find best $\theta$ that maximizes $J(\theta)$
• For the target function $J(\theta)$, which is the quality of a policy $\pi_\theta$, there are 3 candidates: whatever you choose the quality, the policy is always optimized = Let $J(\theta)$ be any policy objective function.
  1. In episodic environments, we can use the start value $J_1 (\theta) = V^{\pi_\theta}(s_1) = \mathbb{E}_{\pi_\theta} [v_1]$
  2. In continuing environments, we can use the average value $J_{av V}(\theta) = \sum_s d^{\pi_\theta}(s)V^{\pi_\theta}(s)$
  3. In continuing environments, average reward per time-step $J_{avR}(\theta) = \sum_s d^{\pi_\theta}(s) \sum_a \pi_\theta (s,a) \mathcal{R}_s^a$
• Policy gradient algorithms search for a local maximum in $J(\theta)$ by ascending the gradient of the policy, w.r.t. parameters $\theta$: $\Delta \theta = \alpha \nabla_\theta J(\theta)$,

 $ \nabla_\theta J(\theta)=$ 

$$\left(\begin{array}{c}\frac{\partial J(\theta )}{\partial {\theta }_1}\\ ⋮\\ \frac{\partial J(\theta )}{\partial {\theta }_n}\end{array}\right)$$ 

🥏 Policy Gradient Theorem

• The policy gradient generalizes the likelihood ratio approach to multi-step MDPs
• For any differentiable policy $\pi_\theta (s,a)$, for any of the policy objective functions $J = J_1, J_{avR},$or $\frac{1}{1-\gamma} J_{avV}$, the policy gradient is 

$\nabla_\theta J(\theta) = \mathbb{E}_\theta [\nabla_\theta (\log \pi_\theta (s,a)Q^{\pi_\theta}(s,a))]$

🥏 Monte-Carlo Policy Gradient: REINFORCE

• Using return $v_t$ as an unbiased sample of $Q^{\pi_\theta}(s_t, a_t)$

$\Delta \theta_t = \alpha \nabla_\theta \log \pi_\theta (s_t, a_t) v_t$

• Psuedo code (Used in Alphago)

https://datascience.stackexchange.com/questions/48872/reinforce-algorithm-with-discounted-rewards-where-does-gammat-in-the-update-c

• Huge variance problem -> Solved by Actor-Critic

🥏 Actor-Critic Policy Gradient: Actor-Critic

• REINFORCE algorithm still has high variance AC is reduce the variance using critic that can use action-value function
• Learn Action-value and policy both
• Actor-critic algorithms maintain two sets of parameters
  • Critic) Updates action-value function parameters $\textbf{w}$
  • Actor) Updates policy parameters $\theta$, in direction suggested by critic
• Actor-critic algorithms follow an approximate policy gradient

$\nabla_\theta J(\theta) \approx \mathbb{E}_{\pi_\theta} [\nabla_{\pi_\theta} (\log \pi_\theta (s,a)Q_\theta(s,a))]$

$\Delta \theta = \alpha \nabla_\theta [\log \pi_\theta (s,a) Q_\textbf{w} (s,a)]$