RL Policy Gradients (REINFORCE → PPO)

Round 1 — Mental Model

Picture training a dog with no manual, only treats. You can't tell it the "correct" action — you can only let it act, observe the outcome, and adjust the odds it repeats what it just did. If a behavior led to treats, make it more likely; if it led to nothing, make it less likely. Crucially you nudge probabilities, not a fixed answer, so the dog keeps exploring. That is REINFORCE: sample behavior, weight each action by the return that followed, push probability toward the good ones.

The problem is noise. One good outcome might be luck; one bad outcome might follow a good action. Naïve REINFORCE reacts to every fluctuation and learns erratically. The fixes are all about judgment: compare each action against what you expected for that situation (a baseline), and refuse to change your whole personality after a single surprising day (a trust region). PPO is REINFORCE that has learned patience.

Round 2 — Internal Mechanics & Mathematical Model

The objective

Let a trajectory be \(\tau = (s_0,a_0,r_0,\dots)\) with return \(R(\tau)=\sum_t \gamma^t r_t\). The policy defines a distribution over trajectories \(p_\theta(\tau)\). We maximize:

\[ J(\theta) = \mathbb{E}_{\tau\sim p_\theta}\!\big[R(\tau)\big] \]

The policy gradient theorem (derivation)

We want \(\nabla_\theta J\), but the distribution we sample from depends on \(\theta\), so we can't just push the gradient inside the expectation. The log-derivative trick \(\nabla_\theta p_\theta = p_\theta \nabla_\theta \log p_\theta\) fixes this:

\[ \nabla_\theta J = \int \nabla_\theta p_\theta(\tau)\,R(\tau)\,d\tau = \int p_\theta(\tau)\,\nabla_\theta\log p_\theta(\tau)\,R(\tau)\,d\tau = \mathbb{E}_\tau\!\big[\nabla_\theta\log p_\theta(\tau)\,R(\tau)\big] \]

Because the dynamics \(p(s_{t+1}\mid s_t,a_t)\) don't depend on \(\theta\), \(\nabla_\theta\log p_\theta(\tau) = \sum_t \nabla_\theta\log\pi_\theta(a_t\mid s_t)\). This gives REINFORCE (Williams, 1992):

\[ \nabla_\theta J = \mathbb{E}\!\left[\sum_t \nabla_\theta\log\pi_\theta(a_t\mid s_t)\,R(\tau)\right] \]

Variance reduction: baselines and advantage

REINFORCE is unbiased but high-variance. A state-dependent baseline \(b(s)\) can be subtracted without introducing bias, because \(\mathbb{E}[\nabla_\theta\log\pi_\theta(a\mid s)\,b(s)] = b(s)\nabla_\theta\!\int\pi_\theta\,da = b(s)\nabla_\theta 1 = 0\). Choosing \(b(s)=V^\pi(s)\) yields the advantage \(A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s)\):

\[ \nabla_\theta J = \mathbb{E}\!\left[\sum_t \nabla_\theta\log\pi_\theta(a_t\mid s_t)\,A^\pi(s_t,a_t)\right] \]

Advantage answers "was this action better or worse than average for this state," which is exactly the signal with least irrelevant variance. This is the actor–critic setup: actor is \(\pi_\theta\), critic estimates \(V\) (often via GAE, generalized advantage estimation, trading bias for variance with a parameter \(\lambda\)).

The trust-region problem and PPO

A single large gradient step can push \(\pi_\theta\) into a bad region from which the collected data no longer represents the new policy — performance collapses. TRPO (2015) bounded the KL divergence between old and new policy exactly but expensively. PPO (Schulman et al., 2017) approximates the same trust region with a cheap clipped surrogate. Let \(r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}\) be the probability ratio:

\[ L^{\text{CLIP}}(\theta)=\mathbb{E}_t\!\Big[\min\big(r_t(\theta)\hat{A}_t,\;\operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t\big)\Big] \]

The min + clip removes the incentive to move the ratio beyond \([1-\epsilon,1+\epsilon]\) (typically \(\epsilon=0.2\)): once an update would help "too much," the objective flattens, so the gradient vanishes there. This is the entire reason PPO is stable without second-order optimization.

Complexity, invariants, limiting cases

Complexity: per update \(O(N\cdot|\theta|)\) for \(N\) sampled steps — cost is dominated by environment rollouts, which is why sample efficiency, not FLOPs, is the RL bottleneck. Invariant: subtracting any action-independent baseline leaves the gradient unbiased — a property the whole variance-reduction toolbox rests on. Limiting cases: \(\epsilon\to\infty\) recovers vanilla policy gradient (no trust region, unstable); \(\epsilon\to 0\) freezes the policy (no learning); baseline \(=0\) recovers REINFORCE (unbiased, high variance); perfect critic \(\Rightarrow\) minimal-variance advantage.

Round 3 — Where It Breaks & Expert Debates

Sample inefficiency is the defining weakness. On-policy methods (REINFORCE, PPO) must throw away data after each update because it's no longer drawn from the current policy — millions of environment steps for tasks a human learns in a few. Off-policy actor–critics (DDPG, SAC) reuse a replay buffer and are far more sample-efficient, but trade away PPO's stability. Which family to use is a live, task-dependent debate; PPO wins when environment interaction is cheap or simulated, SAC when samples are precious.

PPO's clipping is a heuristic, not the trust region it approximates. The clip bounds the ratio, not the KL, so a sequence of clipped updates can still drift the policy far. Later analyses (e.g. "The 37 Implementation Details of PPO," and work showing PPO's gains come substantially from code-level tricks like advantage normalization, value clipping, and orthogonal init) argue the algorithm's real behavior is dominated by implementation choices — a genuine reproducibility concern in RL.

Reward specification is the silent killer. Policy gradients optimize the reward you write, not the one you meant — reward hacking. A network-throughput reward can be maximized by starving latency-sensitive flows; an RLHF reward model can be gamed by verbose or sycophantic outputs. Much of applied RL effort is reward shaping and constraint design, not algorithm choice.

Credit assignment over long horizons. When reward is sparse and delayed, the return \(R(\tau)\) attributes credit to every action in the trajectory, most of which were irrelevant — variance explodes and learning stalls. GAE, reward shaping, and hierarchical RL all attack this, none decisively.

Round 4 — AI × Networks Connection

Policy gradients are the mathematical engine behind RL for network slicing and resource allocation. A slice controller's job — map observed demand and channel state to an allocation of PRBs / compute / power — is exactly a policy \(\pi_\theta(a\mid s)\) over a continuous, high-dimensional action space, which is the regime where policy gradients beat value-iteration methods. PPO's stability is not a nicety here: an unstable update on a live RIC controller degrades a production cell, so the clipped, trust-region-bounded step is what makes on-network learning tolerable at all.

The reward-hacking and constraint problems from Round 3 become SLA problems in the network: a naive throughput reward violates latency SLAs for other slices, so the real object is constrained policy optimization (CPO / Lagrangian PPO) with SLA terms as constraints. And the sample-inefficiency weakness is why RL controllers train on the non-RT RIC against a digital twin / simulator before ever touching E2 control — the offline-train / online-serve split maps directly onto the RIC latency gradient.

Cross-links

Networks · O-RAN architecture → the xApp-conflict problem is a multi-agent PPO setup; the RIC is where the policy is served.

Networks · Network slicing + SLA guarantees → the control target: slice allocation as a constrained policy-optimization problem.

Networks · RAN scheduling algorithms → PF/round-robin are hand-designed policies; RL learns the scheduler's utility weighting.

Pending intersection nodes this unblocks: RL for network slicing & resource allocation, Deep RL for RAN optimization, Reinforcement learning for load balancing.