PPO, One Question at a Time: From Rewards to Policy Updates

type

status

date

slug

summary

1. Why can’t a differentiable reward model directly train the policy?

This question appears explicitly in my note, Why need policy gradient? It is the right place to start because it separates two kinds of derivatives.

A language model produces a distribution. We sample discrete tokens from it. A reward model scores the resulting text:

The reward network may be differentiable with respect to its own parameters or its input embeddings. That does not give ordinary backpropagation a derivative through the choice of a sampled token ID. Changing the probability of choosing “yes” is different from continuously changing the embedding of an already-selected “yes.”

Here is a minimal example. There are two answers: a good answer with reward 1 and a bad answer with reward 0. Let the probability of the good answer be . The expected reward is simply:

The expected reward has a useful derivative even though a particular discrete sample does not provide an ordinary pathwise derivative through the sampling operation.

The score-function identity gives a route to that derivative. Hold the prompt fixed, and assume the reward function is fixed while updating the policy:

The reward supplies a weight. The differentiable path runs through the log probability of the sampled answer. For an autoregressive model, that log probability is a sum of token log probabilities. This is the foundation of the policy-gradient derivation in Spinning Up; its official source is also available on GitHub.

Policy gradient does not replace backpropagation. It tells us what to backpropagate. In code, a basic actor loss looks like -(weight.detach() * log_probability).mean().

The limitation is specific, not absolute. With a tiny action space, I could enumerate every action and differentiate the expected reward exactly. Continuous relaxations offer other estimators. But scoring a weighted average of token embeddings generally does not equal averaging the scores of discrete texts. That changes the problem rather than removing the issue.

2. Reward, return, value, and advantage answer different questions

These quantities are easy to blur together when every one of them is called a “score.” I find it more useful to distinguish their jobs.

Quantity	Question it answers
Reward, rₜ	What feedback arrived after this action?
Return, Gₜ	How much reward was accumulated from here onward?
Value, Vπ(sₜ)	Before choosing an action here, what return should this policy expect?
Action value, Qπ(sₜ,aₜ)	After choosing this action, what return should this policy expect?
Advantage, Aπ(sₜ,aₜ)	How much better or worse is that action than the policy’s average at this state?

Formally, . A sampled return minus an estimated value is an estimator, not the exact advantage function.

Suppose a completed answer earns 8 points. At a prefix where the expected eventual score was 7, its Monte Carlo advantage estimate is +1. At a prefix where the expectation was 9, the estimate is −1. A positive reward can produce a negative advantage. “Negative” means below the chosen baseline, not necessarily objectively bad.

Why can we subtract a baseline without changing the expected policy gradient? For a fixed state and an action-independent baseline held fixed in the actor update:

A useful baseline can reduce sampling noise without changing the expected direction. An arbitrary baseline is not guaranteed to reduce variance, and a learned value function is not automatically the variance-minimizing baseline. The important distinction is between the zero-mean identity and the practical quality of the estimator. Spinning Up explains the baseline identity and value-function approximation together.

What a negative advantage does to the logits

For one selected token , consider the loss . With softmax logits , ordinary differentiation gives:

For positive advantage, gradient descent pushes the selected logit up and the others down. For negative advantage, the directions reverse. This explains why learning can use unsuccessful or below-baseline samples rather than only imitating successful ones.

There are two limits to that intuition. First, it describes one sample’s contribution at the logit level. Shared parameters, other samples, and optimizer state affect the final update. Second, lowering a failed answer’s probability does not identify the correct replacement. The reward is not secretly providing a full target answer.

3. What does the critic predict, and where does GAE enter?

For token-level language modeling, the state before an action is the prompt plus the prefix already generated:

An outcome reward model scores a completed response. A critic estimates the expected future return from a prefix under the policy being evaluated. It is a forecast, not another name for the final reward.

The next state is the prefix with one more token appended. That transition is deterministic, but it is still a state transition. Alternatively, I can treat the whole response as one action in a contextual bandit. These are different modeling resolutions. “RLHF is always a bandit” and “language generation has no transition function” are both too broad.

My questions about GAE were really questions about how to combine observed outcomes with those forecasts. With rollout-time value estimates, define a temporal-difference residual:

Here, means the action ends a true terminal episode. GAE accumulates these residuals backward:

The original GAE paper, Section 3, develops this as a weighted mixture of multi-step estimators. At , it is a one-step TD residual. At , over a complete episode with a terminal value of zero, the intermediate values telescope away and it becomes return minus the initial value estimate. Intermediate settings trade reliance on sampled future outcomes against reliance on the critic; an inaccurate critic can introduce bias through bootstrapping.

A three-step example

Take rewards (0, 0, 1) and value estimates (0.2, 0.4, 0.5, 0), where the final zero is the terminal value. With and :

Working backward gives:

These are discounted sums of TD residuals, not merely discounted sums of rewards. The nonterminal residuals can be nonzero even when their immediate rewards are zero.

For a common GAE-based critic target, add back the rollout-time value:

The targets in this example are (0.895, 0.95, 1.0). Train the critic toward those fixed targets with a squared-error loss. Use the raw advantages to construct the targets; normalizing advantages for the actor does not turn the normalized numbers into valid value targets. Other implementations may use Monte Carlo return targets or additional value clipping. Those are choices to inspect, not interchangeable labels.

In the simple implementation I would build first, rollout log probabilities, rollout values, advantages, and critic targets stay fixed during the batch’s optimization. The current actor and critic predictions change. Actor advantages are detached, so the actor loss cannot “improve” by changing its own weights instead of the policy.

This also resolves my earlier question about Bellman equations. Using sequence-level Monte Carlo policy gradients does not require explicitly performing a Bellman backup. Token-level PPO with TD residuals and GAE does use the value-function recursion. Bellman structure has not disappeared; the estimator determines how much of it is used.

GAE estimates a learning signal. It does not prove that a particular token caused the final success. With only an outcome reward, causal credit assignment remains difficult. Also, a rollout cut short by a collection limit is not automatically a true terminal state; its bootstrap treatment must match the task.

4. If the data is on-policy, why does PPO need importance sampling?

This is the exact question I asked in one of the retrieved discussions:

In PPO, why do we need to do importance sampling if the data is collected on-policy?

The missing variable is time. The data was on-policy when it was collected. PPO then reuses it while the policy changes. The original PPO algorithm alternates rollout collection with multiple optimization epochs over that batch.

Let be the policy that generated the batch. During optimization, compare the current policy against the saved behavior probability:

At the start, the policies match. After one optimizer step, the current numerator changes while the denominator still describes how the data was generated. An action sampled with probability 0.10 may now have probability 0.13, giving a ratio of 1.3.

At a fixed state, the importance-sampling identity is exact, assuming the behavior policy has support wherever it is needed:

But there is an important boundary. A token-level action ratio does not by itself correct the entire distribution of trajectories or prefixes. PPO’s local surrogate still uses states visited by the old policy and estimates associated with that rollout. Full trajectory reweighting would involve additional ratios. This is one reason to keep the update local rather than treat old samples as an unlimited replay buffer.

A ratio of one is not a constant function

Initially, . Nevertheless:

The ratio’s value is one at that point; its slope need not be zero. Replacing it with the literal constant 1 destroys the gradient. Differentiating through an identical numerator and denominator also destroys the intended correction. The denominator must be the fixed behavior probability.

This gives a precise answer to the related “online rollout does not need clipping” question. With exactly one fresh, matching full-batch gradient evaluation, the ratios initially equal one, so clipping is inactive. That does not guarantee the resulting finite optimizer step is small. Several minibatch steps are already several updates, even inside one epoch. Asynchronous or otherwise stale rollouts may not start at a ratio of one at all.

5. What does PPO clipping actually remove?

Here I mean PPO-Clip, rather than PPO’s alternative KL-penalty formulation. The objective to maximize is:

The min is the central operation. Clamping a ratio without that minimum is not the same objective. The original paper’s Equation 7 and Figure 1 show that clipping depends on the advantage’s sign.

For , consider four individual samples:

Advantage	Ratio	Selected objective term	Gradient of this term with respect to current log probability
+2	1.3	min(2.6, 2.4) = 2.4	0: already increased enough
+2	0.7	min(1.4, 1.6) = 1.4	+1.4: still asks to increase probability
−2	0.7	min(−1.4, −1.6) = −1.6	0: already decreased enough
−2	1.3	min(−2.6, −2.4) = −2.6	−2.6: still asks to decrease probability

These are gradients of an objective being maximized. A gradient-descent loss uses its negative.

For a positive advantage, PPO stops this sample from rewarding further increases beyond the upper threshold. For a negative advantage, it stops this sample from rewarding further decreases below the lower threshold. Movement in the wrong direction still makes the objective worse.

The interpretation I want to retain is: “Do not let this old sample keep claiming additional benefit from an already-large favorable change.”

That is not a hard probability constraint. The ratio can cross the threshold. Other samples, shared parameters, a critic loss, entropy regularization, and optimizer momentum can keep moving it. Clipping also does not impose a universal KL bound or guarantee that actual task performance improves. The Spinning Up PPO documentation, also available in its official source, makes the need for additional safeguards such as KL-based early stopping explicit.

6. Why is there another KL term in RLHF?

My notes contain two different ideas called “stay close to the old model.” They should not share a name without specifying the anchor.

Anchor	Typical lifetime	Purpose
Rollout policy, πold	Refreshed for a new rollout batch	Reference point for the local update and probability ratio
Reference policy, πref	Often fixed across an RL training stage	Regularize cumulative departure from a chosen model, often the SFT checkpoint

Small steps from yesterday’s position can accumulate into a large distance from the starting point. That is why rollout-relative control and reference-relative regularization are not redundant.

A common sequence-level RLHF objective is:

The RLHF Book’s regularization chapter explains this reference-policy role. Implementations can incorporate a KL-related term into rewards or introduce an explicit regularization loss. The estimators, sampling distribution, and gradient paths still need to match the intended objective; casually adding both can double-count the penalty.

For a fresh sample, a token log-ratio to the reference is a sampled contribution, not a full vocabulary KL by itself. An individual log-ratio can be negative even though the corresponding KL expectation is nonnegative. The sequence log-ratio is the sum over generated tokens.

A reliable verifier may change the motivation for a reference penalty, but it does not logically make regularization unnecessary. Conversely, reference KL is not mandatory in every RL setup. For example, MiniMax-M1’s CISPO formulation omits that penalty. Removing reference KL and removing PPO clipping are separate decisions.

7. PPO versus CISPO: moving a factor changes the gradient path

Another discussion focused on how the probability ratio can be “taken outside” the derivative. With fixed advantage and behavior probabilities:

Therefore, the following expression produces the same first derivative at the evaluation point:

This is a statement about a gradient implementation. It is not equality of scalar objective values, and it does not imply identical second derivatives. Without stopgrad, the product rule introduces an extra term.

CISPO, introduced in the MiniMax-M1 report, Equations 4–5, clips the importance weight and detaches it before multiplying by the token log probability:

The report uses group-relative advantages and token-level aggregation. Its experiments leave the lower bound inactive. The essential contrast is that an oversized importance weight is capped rather than making the entire favorable token contribution flat.

For an illustrative upper cap of 1.2, a sample with advantage +2 and ratio 1.3 has zero PPO-Clip gradient from that term. The CISPO-style term has a derivative of +2.4 with respect to its current log probability. This cap is a teaching example, not a recommended paper hyperparameter.

Preserving a contribution does not automatically make an algorithm better. Weight clipping introduces bias; it also does not bound the norm of the entire parameter gradient. The useful habit is to compare computational graphs, not just look for the word clip in two formulas.

8. Rebuilding PPO from scratch: what I would implement first

I would not start with four large models and a distributed rollout engine. I would first build something small enough that the correct answer is visible.

Start with a two- or three-action bandit. Enumerate its exact expected reward and gradient. Check that the expected score-function estimator matches. Then add a baseline and confirm that the expectation stays the same.

Add a short sequential problem. Give it a terminal reward and explicit value estimates. Compute returns, TD residuals, GAE, and critic targets by hand. The three-step example above is enough to catch an indexing mistake.

Freeze a rollout batch. Save the behavior log probabilities and targets. Take several policy updates on that batch. Recompute only the current probabilities. Observe when the ratios leave one, then add the clipped objective and test all four sign-and-ratio cases.

Only then add a language model. This introduces important indexing and masking work, not a different score-function identity. The position predicting a token must align with that token’s label. Select the sampled token’s log probability from the vocabulary distribution. Exclude prompt and padding positions from the response loss, but include a genuine generated EOS action. Match the probability calculations to the actual rollout sampling distribution.

A minimal training loop is:

Policy, behavior policy, reference policy, critic, and reward scorer are roles, not necessarily five separately resident large models. Saved behavior log probabilities can avoid another full behavior-model copy; actor and critic may share a backbone; a verifier can replace a learned reward model. The PPO paper’s actor–critic formulation allows shared parameters, while the RLHF reproduction literature shows how much implementation choices matter in language-model training.

The expandable code below checks six mathematical properties. It runs on CPU with PyTorch and needs no model download. These checks were executed while preparing this draft. They are not a PPO trainer or evidence of a training benchmark result.

Runnable PyTorch checks: gradient identity, ratio=1, directional clipping, CISPO-style weighting, and GAE

Save this code as ppo_sanity_checks.py in an environment with PyTorch installed, then run python ppo_sanity_checks.py. Expected output: All six PPO sanity checks passed. The GAE helper deliberately handles one complete, unpadded episode; a production implementation needs explicit masks and truncation handling.

9. What can fail even when the loss looks correct?

My debugging priority would be to test invariants before tuning hyperparameters. The ICLR blog post on implementation details is useful here because it follows a reproduction rather than presenting only a loss equation.

A mismatched behavior probability can invalidate the ratio. Before any update, reevaluating a fresh rollout under matching conditions should give ratios near one. Temperature, truncation of the sampling distribution, dropout, precision, and token alignment can break that expectation. A top-k or top-p sampler also changes support; it is not enough to pretend that unrestricted model probabilities were the behavior distribution.

A wrong mask can train on the wrong problem. Prompt tokens and padding should not accidentally become generated actions. A generated EOS token should not disappear merely because a masking rule confuses it with padding. An early collection cutoff should not silently become a terminal success or failure.

A critic can provide a confident but poor forecast. Inspect target scale and held-out prediction behavior, not just the scalar value loss. A declining MSE can coexist with unhelpful advantages if the targets or the data distribution are wrong.

A good-looking actor loss can coexist with worse outcomes. A surrogate is not the actual expected task reward. Track reward on newly generated samples, independent task quality, response length, and diversity. Also separate divergence from the rollout policy from divergence from the reference. For clipping statistics, distinguish “ratio outside the interval” from “this term is flat under the sign-dependent clipped objective.”

Finally, none of these checks makes a bad reward function good. Stable optimization can still optimize the wrong proxy. PPO is not a substitute for an independent evaluation set or a reward-design review.

10. Which learning materials are worth returning to?

I would organize the reading around the question each source resolves, not around collecting more tutorials.

The best mathematical backbone: Shiyu Zhao

Mathematical Foundations of Reinforcement Learning — author’s book and lectures is the backbone already emphasized in my reading plan. Its progression connects Bellman equations, Monte Carlo estimation, stochastic approximation, TD learning, policy gradients, and actor–critic methods.

For this particular topic, I would return to the policy-gradient and actor–critic chapters after refreshing value estimation. The output should be one connected derivation, not another completion checkbox. PPO clipping and GAE still deserve their own primary-source treatment; the book is the foundation, not a substitute for those papers.

The best short route through PPO: Spinning Up, then the paper

Read Intro to Policy Optimization, then PPO, then Sections 2–5 of Proximal Policy Optimization Algorithms. Use the linked official GitHub sources above when the documentation site is unavailable.

My completion test would be to draw the two clipping curves from memory and explain why the minimum behaves differently for positive and negative advantage. After that, read Section 3 of the GAE paper and reproduce the telescoping sum.

The best connection to language-model post-training: RLHF Book

Nathan Lambert’s policy-gradient chapter and regularization chapter connect the derivation to language models. I would read them with a specific question: which objects belong to the RL objective, which belong to an estimator, and which belong to the training implementation?

The best implementation reality check: N and N+

Two similarly named resources deserve different labels. The N Implementation Details of RLHF with PPO studies reproduction details around earlier language-model preference training. The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization is the separate summarization reproduction.

I would use them to build a checklist for log probabilities, reward placement, value targets, normalization, EOS handling, and optimizer behavior. Their settings are evidence about particular experiments, not universal PPO defaults.

The best next exercise for LLM engineering: CS336

The official CS336 alignment assignment is a useful bridge from these ideas to a tested implementation. The current repository includes GRPO tests and an optional safety/RLHF supplement. It is not a complete token-level PPO-with-critic assignment, so I would not treat completing it as proof of implementing all of PPO.

My saved copy of Cameron Wolfe’s PPO for LLMs: A Guide for Normal People is a supplementary orientation resource. I would use a narrative overview to locate questions, then settle gradient behavior with original equations and executable checks. A readable overview and an implementation authority serve different purposes.

For understanding PPO itself, my shortest route is Spinning Up → PPO paper → GAE paper. For connecting it to work on LLMs, add RLHF Book and one reproduction. Zhao’s book remains the reference for rebuilding the underlying mathematical structure.

11. What transfers beyond PPO?

GRPO, introduced in DeepSeekMath, makes one transfer especially clear. It estimates a relative signal from a group of responses instead of relying on PPO’s usual learned value baseline. That changes how the learning weight is obtained; it does not remove the need to understand sampling, probability ratios, clipping, or the limitations of sequence-level credit.

The same separation helps with an agent that takes tool actions. The outcome might be successful task completion, the state might include tool observations, and the actions might be API calls. The questions remain: where did the data come from, what is the target objective, how is the learning weight estimated, and what limits a misleading update?

The most confusing step in this whole chain is the distinction between a scalar objective and an expression designed to produce its gradient. A detached reward times a log probability is not the expected reward itself. A ratio equal to one can still have a derivative. A detached importance weight can preserve a first-order gradient while changing higher-order behavior.

That is the understanding I want to make durable: do not stop at recognizing the formula. Identify the sampling distribution, circle the quantities held fixed, and take the derivative. Then test a case where the intuitive story could be wrong.

Editorial appendix: source notes and discussion trail

This appendix records the basis for the draft. It is not part of the public-facing argument and should be reviewed before publishing. Chat retrieval returns relevant excerpts, not a complete export of every conversation; dates below follow the retrieved records in UTC. The outline is organized by concepts, not presented as proof that every topic has been mastered.

Notion notes used

Why need policy gradient? — differentiable reward models versus discrete sampling; exact expectation versus a sampled estimator; the core starting question.

🏟️

RLHF Book — policy optimization, RLHF versus conventional RL, and the sequence-level bandit view.

Post training, including its linked notes , , and — broader questions about outcome optimization, online sampling, and credit assignment. These inform the context without turning this article into a survey of every post-training method.

📕

01 · 强化学习的数学原理 and — the existing mathematical study path. Both records were marked Done when checked, but the derivation and implementation deliverables in the reading-plan body remained unchecked. A status field is not evidence that those deliverables were completed.

PPO for LLMs: A Guide for Normal People — saved explanatory article; its reading status was Not started when checked. The draft does not claim it was already completed.

Stanford CS336 Language Modeling from Scratch — an In progress reading record without substantive body notes; used as evidence of interest, not completed coursework.

Questions recovered from discussions

Date	Retrieved question or discussion topic	Where it appears here
2026-05-03	How RLHF relates to standard RL; contextual bandit versus token-level MDP; local versus reference-policy regularization	Sections 3 and 6
2026-08-25	GAE versus rewards; detached token advantages; positive and negative learning signals	Sections 2 and 3
2026-08-26	How rewards become per-token advantages; what the critic target is; which policy produces a rollout	Sections 3 and 4
2026-08-28	“In PPO, why do we need to do importance sampling if the data is collected on-policy?”	Section 4
2026-08-30	PPO versus CISPO; how the ratio factors out of the gradient	Section 7
2026-08-31	Why RLHF descriptions sometimes appear not to use Bellman equations	Section 3
2026-09-03 to 2026-09-04	Behavior versus target policies; on-policy versus off-policy learning; connecting RL’s mathematical pieces	Sections 4, 8, and 10

The current chat-history context also contains September 14 question titles about fresh online GRPO rollouts and negative-gradient learning. Only their visible questions were available here; no unseen answers are attributed to those conversations.

Clarifications added during this synthesis

The draft distinguishes deterministic transitions from no transitions, a locally correct action reweighting from a full trajectory correction, clipping from a hard trust-region guarantee, a reward model from a value function, and GAE from causal token attribution. It also separates the original discussion topics from new numerical examples and newly executed checks. No prior large-scale experiment, benchmark result, or professional experience is inferred from the notes.