πΊ PODCAST β Nathan Lambert's Post-Training Lectures
Rating: 10/10 | Intermediate to Advanced AI Scientists/Engineers
πΌ "Of all existing things, some are in our power, and others are not in our power. In our power are thought, impulse, will to get and will to avoid, and, in a word, everything which is our own doing. Things not in our power include the body, property, reputation, office, and, in a word, everything which is not our own doing." - Epictetus (c. 100 CE)
This week, I looked at the very insightful and fresh series on post-training by Dr. Nathan Lambert.
I discovered his work early last year, and since then, I have been following his writing. With the work I have been doing, I found the timing of these lectures very serendipitous.
He also just released a book on RLHF, which is in my reading list and will certainly become a post eventually.
For anyone new here, post-training is directly relevant to the work we're doing on NULLSTAR's PleiadesOS:MEROPE module.
As I read through the material, below are my highlights β with a moderate attempt at coherence β for my personal inventory (and perhaps as a way to connect with other friendly AI scientists and builders out there who love talking about the same things). As always, please remember our rules:
π€ If you see a πΎ, it means I found something particularly cool or learned something new.
π€ If you see a β¨, it means things that are so cool that they need a lil glitter around them.
π€ If these notes look interesting to you, it's your β¦moral dutyβ¦ to watch the videos..
π€ Just another blog, but not your usual blog: they're living notes that I change as I fancy.**
πΆ Lecture 0 β‘οΈ ML Foundations for Post-Training
A language model learns the joint probability of a sequence of tokens in an autoregressive manner β each prediction depends on the token before it.
- Modern LLMs are decoder-only Transformers built on self-attention (each position attends only to itself and preceding positions).
- The LM head β the transformer backbone outputs a hidden state vector per token β the head is a final linear projection from the hidden dimension back to the vocabulary (the tokenizer space).
- Hidden state β LM head β logits (one score per vocab token).
- The same backbone can carry different heads for different jobs β reward-model head: same transformer, new head, new objective.
- Softmax turns the logits at step
tinto the next-token distribution, producing each per-token probability β a whole sequence is the product of these terms β a sum in the log space β every term in the sum is alog_softmaxof the logits at the true token, which is the (negative) cross-entropy at that position.
- Masking β three different types: 1) causal mask (built into decoder attention), 2) attention / padding mask (tells attention to ignore padding tokens), 3) completion / loss mask (
1on completion tokens, prompt tokens contribute no loss).
- Sampling generation from the distribution during decoding β same weights but many ways to pick each token: 1) greedy; 2) beam search; 3) temperature; 4) truncated (from top-
k); 5) lookahead search (MCTS).
- Cross-entropy loss β compares the modelβs predicted distribution to the true next-token label.
- Training is measured in optimizer steps / tokens; epochs are a useful unit for finite fine-tuning datasets β the learning-rate schedule matters as much as the rate itself.
- The same backpropagation loop is shared across pre-, mid-, and fine-tuning β what differs is masking, sampling, batching, data construction, and systems (not the optimizer).
- Mid-training β the annealing / high-quality-data phase at the very end of pre-training, before any instruction data β modern post-training pipelines start here.
KL (Kullback-Leibler) divergence β measures how one distribution differs from another. Itβs asymmetric, non-negative, and keeps the policy close to its SFT start. The entropy (similar equation on one distribution) is the uncertainty of the distribution, while the cross-entropy is the equation on two distributions.
Cross-entropy(p, q) = Entropy (p) + KL_Divergence(p, q)
- The sum behind KL is intractable, but since KL is an expectation, we can estimate it with Monte Carlo (average the log-ratio over samples drawn from the policy).
- A preference is a binary outcome, so a score difference is squeezed through a sigmoid into a probability β only relative reward differences matter.
The goal of RL is to choose policy parameters that maximize expected reward under the policy itself. A human preference is a single scalar at the end of a response β policy gradients optimize the expected evaluator score anyway, first by learning that signal as a reward model, then by optimizing it β the whole motivation for RLHF: change the modelβs distribution over responses toward behavior we want.
πΆ Lecture 1 β‘οΈ Post-Training and RLHF Overview
- InstructGPT β made the classic 3-stage recipe canonical:
1. Instruction fine-tuning (IFT/SFT)
β foundations β teaches the model to follow instructions (the format)
2. Reward model
β learns to score quality from human preferences (the signal)
3. RL against the reward model (re-use preference prompts, PPO optimization)
β aligned model: uses those scores to curate better training data (the optimization, which will be replaced with RL)
- Preference tuning/training landscape β optimize the same underlying objective but differ in how they move the policy toward higher-reward completions β debates on performance β we donβt have a reward function β the reward model is a proxy, not ground truth (the well-trained RM is corrected with the real user choices, i.e., Goodhartβs law), so overfitting is real.
- PPO (online RL)
β generate, score, update policy
- Rejection sampling
β fine-tune on the best samples (filter, then SFT)
- DPO
β direct gradient for RLHF on preferences (no need to train an extra reward model)
- In practice:
- Sycophancy
- Reward hacking
β RM score climbs, but quality degrades
- Verbosity bias
β longer responses score higher
- Over-refusal
β model refuses legitimate queries
- KL penalty helps (limits how far the policy can drift from the reference model)
β over-optimization is a problem
- From RLHF to post-training β evolving into a complex series of steps to get the final, best model.
- Base model β aligned model
Nβ reward model / LLM judge β aligned modelN+1β final model.
- Modern systems keep the same core idea of using multiple optimizers with different strengths and weaknesses, but add more stages, more data, more filtering β and then tools like RLVR and model merging.
- Frontier β RL with verifiable rewards β RLVR is much simpler than RLHF because it removes the reward model and replaces it with a verifiable reward β unlocked inference-time scaling β more compute at generation time per problem β a whole new scaling law β o1 and DeepSeek-R1 β notions of agentic models acting.
- o1 β test-time scaling β how many tokens you spend on the problem β a log-linear relationship between inference compute (number of tokens generated) and downstream performance:
- Fundamental property of models, unlocked with RLVR
- Can be done in many ways
β one long chain-of-thought sequence, multiple agents in parallel, or a mix
- Improving inference-time scaling changes the slope and offset of the curve
πΆ Lecture 2 β‘οΈ RLHF Foundations, IFT, Reward Modeling, Rejection Sampling
- Instruction tuning emerges from two parallel research threads:
1. Unified text-to-text framework
β T5 framing every NLP task as βtext in, text outβ worked well
2. Scaling + instruction following
β FLAN, T0, and natural instructions
β training on diverse tasks with explicit instructions improved zero-shot generalization
3. SFT took off after ChatGPT, Alpaca, OpenAssistant, TΓΌlu
β turned instruction tuning into a broadly reproducible open recipe
- Chat templates β the structure of instruction tuning β early chat templates defined 3 roles β system (background instruction, persona, constraints), user (the human's messages), assistant (the modelβs response) β the model generates until it produces an end-of-turn token β snippets of Jinja code stored in the tokenizer config.
- Prompt masking β during SFT, predict assistant responses, not user messages:
- System and user tokens β masked from the loss
- Only assistant completion tokens contribute to gradient updates
- The model learns how to respond, not how to ask
- For multi-turn conversations:
1. Final-turn only
β mask everything except the last assistant response
2. All assistant turns
β mask only user/system tokens, train on every assistant response
- Best practices for instruction tuning:
- Data quality matters more than quantity:
β High-quality completions are critical
β prompts are masked anyway, model learns from responses
β ~1M prompts is sufficient for good RLHF-ready models
β SFT prompt quality has DECREASED with reasoning models
- From pre-training:
β Batch size much smaller
β Learning rate 1-2 orders lower
β Loss function β same cross-entropy, but only on unmasked assistant tokens
- Data scaling for IFT β The amount of instruction data needed has evolved rapidly:
- Early post-ChatGPT
β ~10k high-quality human-generated samples could be SOTA
β Now, large-scale synthetic datasets, quality filtering is essential
- Before
β scaling the prompts enabled more performance
β Now, RMs use more compute and tokens to train, via more tokens per prompt in SFT (and longer context lengths)
- Building SFT data β synthetic completions:
- Synthetic data has become the dominant approach for building SFT datasets (Wang et al., 2023)
- Start with N high-quality (human-written) prompts
- Ask a strong LM to create a modified version
- Generate completions with another strong LM
- 10x more training data
- Human data still needed for out-of-distribution or novel tasks (knowledge work)
- The role of reward model in RLHF:
- The reward model plays the role of the environment
β it returns a reward signal that tells the policy how well it did
- Difference with RL
β in RLHF, we get to control and learn the reward function from human preferences, not FIXED by the environment
- Reward model compresses complex subjective human judgments into a single scalar score
The Bradley-Terry model of preferences
- A probability model is a mathematical form that assumes it matches how real judgments work β then we fit its parameters to data β canonical reward from BT model, 1952.
- Given two items
iandj, represents the probability that the judge prefersioverjβ each item has a latent strength, which can be reparameterized using exponentials to work with unbounded scores β shift the probability function to a sigmoid function between two scalars (the difference of rewards foriandj).
- Only score differences matter β Binary cross-entropy β the reward model is learning to classify which completion was preferred.
- Implemented similarly to a transformer but instead of having the language modeling head, which outputs logits per token in the vocabulary, it appends a linear head that outputs a single scalar (goes from the hidden size of the language model to a dimension
1):
- Whatβs the probability that that is a chosen response?
β Itβs a latent score/power of the input text
- Gets the reward from chosen and rejected and then calculates the loss
- Reward models are typically trained for only 1 epoch to avoid overfitting to the preference data
- Preference margin loss β When annotators provide Likert scale ratings, the magnitude of the preference can inform training β encourages the model to produce larger score gaps for strongly preferred pairs.
k-wise loss and balancing comparisons β InstructGPT balances multiple comparisons per prompt to prevent overfitting (also included all prompts in the same batch at training time).
k-wise loss β handles full rankings overkcompletions (k = 2reduces to Bradley-Terry).
- Outcome reward models (ORMs):
- For reasoning tasks, we have verifiable correctness
β ORMs train a per-token head using the completion-level correctness label repeated across completion tokens
- Key differences from Bradley-Terry
β no pairwise comparisons needed, just correct/incorrect labels per response
β the model outputs a per-token probability of correctness, not a single score at the EOS token
- Score at the last token vs. score at every token
β both start from the same base LM hidden states, the difference is where the head is applied
- Outcome reward model is not the same as a Bradley-Terry model trained on correct vs. incorrect comparisons β different architectures:
- A BT model trained on correct/incorrect pairs is still a preference RM
β uses correctness as the preference signal
β a true ORM has a fundamentally different input-output structure
- Process reward models (PRMs)
- Score intermediate reasoning steps, not just final outcomes
- Labels are applied only at step boundaries; other tokens are masked during training
- Generative reward model β LLM-as-a-judge
β An alternative to training a reward model β prompt an LLM to judge quality
β Benchmarks β MT-Bench, AlpacaEval, Arena-hard, WildBench
β Generative RMs tend to UNDERPERFORM trained reward models on RM evals, but are cheaper
β Very useful on synthetic data pipelines
- Rejection sampling β Simplest thing you can do with the RM for downstream training:
1. Generate many candidate completions
2. Score them with a RM
3. Keep only the best ones
4. Fine-tune on those:
β No policy gradients, no online RL
β just filtered supervised learning
β Offline data curation
β generate first, then train on the filtered outputs
- Need to be careful with sampling settings β core hyperparameters:
- Temperature β 0.7-1.0; need diversity in completions
- Completions per prompt β 10-30+ (too few, noisy selection)
- Fine-tuning
β standard SFT on selected completions (same loss, but details like learning rate may differ from initial IFT)
- TIP: sort completions by length before batch RM inference to reduce padding token computation
β reward model batches can be about length
- How to sequence RS in a multi-stage pipeline, whether to use generations from multiple models, optimal prompt selection? There hasnβt been a fully open reproduction of this, which is confusing.
- Best-of-N sampling β rejection sampling without fine-tuning:
- Same procedure, but skips the fine-tuning step
- Used at inference time to pick the best completion
- Does not modify the model β itβs a sampling technique
- Often used as a baseline comparison for online RL methods like PPO
- Simplest possible reward-guided method
β generate more, pick the best
β can also be done with verification / LLM-as-a-judge
- The pipeline so far:
1. Instruction fine-tune the retained model β learn the chat format
2. Collect preference data β human annotators compare pairs of responses
3. Train an RM on preference data β learns to score quality
4. Rejection sampling β generate many completions, keep the best, fine-tune
πΆ Lecture 3 β‘οΈ Policy Gradient Algorithms for RL on LLMs
Overall, RL losses on language models are robust, scalable, effective, and flexible, opening large new fields of experimentation.
- RLHF β reward model scores subjective quality; RLVR β verification function instead of a reward model that checks correctness (math, code) β same policy gradient algorithms, different reward source.
- Notation:
β‘οΈ (s, a) for state and action <β (x, y) for prompt and completion
β‘οΈ state is the prompt + tokens generated so far
β‘οΈ action is the next token y_t
β‘οΈ r_t for per-step rewards
β‘οΈ R(tau) for trajectory returns
β‘οΈ rho_t for importance-sampling ratios
β‘οΈ pi_theta(a_t | s_t) is the next-token distribution
β‘οΈ episode is the one prompt β completion
β‘οΈ transition is deterministic: append token to sequence
β‘οΈ terminal reward: RM score or verifier output
β‘οΈ gamma β typically 1.0 (no discounting)
β‘οΈ the unit of optimization is the full completion
- What is optimized? Choose policy parameters that maximize reward on average under the current policy β we estimate this expectation with sampled rollouts β we craft a gradient/derivative that lets us optimize this.
- Policy gradient β make actions more likely when they lead to better outcomes β the gradient of the probability tells what actions matter β
psi_tis the learning signal telling the optimizer how good the action was and determines the algorithmβs variance, bias, and compute cost.
- Key RL qualities:
- value functions V(s) β expected future return from state s
- action-value Q(s, a) β expected return after taking action a in s
- advantage A(s, a) = Q(s, a) - V(s) β how much better is an action a compared to average
- Popular choices for
psi_tβ total trajectory reward, future return fromt, baselined return, state-action value function, advantage function, TD residual.
Derivation of the policy gradient
- The reference model is a frozen copy of the policy at the start of RL training (the SFT checkpoint) β it anchors the policy so it doesnβt drift too far during optimization.
- KL divergence as regularization β the KL penalty measures how far the current policy has drifted from the reference AT EACH TOKEN.
- In RLVR, KL is often reduced or removed β the reward is the ground truth, so there is less to exploit β some setups (e.g., DeepSeek R1) drop the reference model, saving memory. β as reward signals become more reliable, the need for KL regularization decreases.
REINFORCE
- The simplest instantiation of the policy gradient β the Monte Carlo form of policy gradient: sample trajectories, compute their returns, use those sampled returns to weight the log-prob gradients. β needs a baseline to reduce variance β no value function required.
The baseline problem in REINFORCE β without a baseline, the gradient weights each action by its raw return, which doesnβt tell you whether an action was better or worse than expected, just how good the overall outcome was β the gradient is high variance because the raw return mixes the quality of the action with the quality of the state β a good action in a bad state and a bad action in a good state can produce similar returns.
- Common baselines β average reward over the batch, moving average of recent rewards, learned value function.
Actor-critic methods β thereβs a learned model which is the policy, and a learned value model which is a critic, critiquing how good each state is.
- REINFORCE RLOO (leave one out) β generate
kcompletions per prompt and use the otherk - 1rewards as the baseline β per-prompt baseline capturing prompt difficulty β iterate on the baseline β make sure the baseline is helping reduce the variance effectively so the gradient signal is useful β per-prompt baseline that naturally captures prompt difficulty - hard prompts get low rewards across all completions, so the baseline is low.
PPO
- All the algorithms so far are on-policy β they generate fresh rollouts from the current policy each batch, then update on rollouts β this is in contrast to off-policy RL methods (e.g., DQN, Atari, old DM stuff), that store and replay old experience β the problem: vanilla policy gradient is sensitive to step size. Too large an update and the policy can collapse, too small and training is painfully slow. TRPO solved this with a hard trust-region constraint, but required expensive second-order optimization. β proximal policy optimization (PPO) gets TRPO-like stability with a simple clipped objective, and because the clipping keeps updates conservative, you can safely take multiple gradient steps per batch of rollouts, improving sample efficiency.
- Core idea 1οΈβ£ β constrained updates β large gradient steps can destroy the policy (instability, over-optimization, etc.) β the solution: trust regions, limit how far the policy can move in a single update.
- What can we do with more conservative gradients β extract more signal from the batch β introduce new problems β how do we constrain the updates over multiple gradient updates? β how do we take the policy gradient if the data has drifted off-policy?
- Core idea 2οΈβ£ β importance sampling β we want to take multiple gradient steps on a batch, but the data came from an old policy β define the importance sampling ratio which tells how the current and old policy agree on the action (which one assigns higher probability) β the ratio reweighs old-policy samples to estimate new-policy gradients.
- The surrogate objective β plugging the importance sampling into the policy β intermediate problem β without constraints, maximizing this can take arbitrarily large steps, the ratio of importance sampling can diverge from
1β PPO introduces the clipped objective.
- PPO clips the ratio to prevent large updates with an epsilon cutoff around
0.1-0.2β a practical surrogate inspired by trust-region ideas
- The value function (critic) β PPO trains a value function alongside the policy β the expected future return from state
s_tβ separate parameters, predicts post-KL-shaped future return at each token position, trained via MSE against post-KL returns β the value function serves as a learned baseline for advantage estimation.
- The critic learns to predict returns β the total discounted reward from each token onward β GAE gives us the advantages by combining actual rewards with the criticβs own predictions.
- Value function initialization is hard β needs to produce reasonable estimates from the start β 1) initialize from RM backbone (value predictions start near actual rewards); 2) initialize from SFT model + random head (cheaper but early training unstable); 3) cold start issues (if initial value estimates are bad, GAE advantages are noisy, early training can be chaotic).
- How PPO-RLHF gets token-level credit (i.e., turns a sequence-level reward into token-level training) β PPO-style RLHF often starts with one scalar reward for the whole completion β how does that become a per-token training signal β 1) KL penalty shapes intermediate tokens, at each token
t, subtractbeta . KL_tfrom the reward β per-token reward signal; 2) Final token gets the RM score:r_T - R(tau) - beta . KL_T, all other tokens getr_t = - beta KL_t; 3) GAE propagates credit backward: the value function + TD residuals assign per-token advantages from these shaped rewards.
- Advantage estimation β it helps by: 1) centering reduces variance (asking how much better is this action than average? How good was the total return?), 2) credit assignment (with per-token values, each token gets its own advantage signal) β Monte Carlo estimate is simple and unbiased β but it can be high variance β temporal difference (TD) methods and generalized advantage estimation (GAE) trade some bias for lower variance.
- Advantage whitening β normalize advantages to zero mean, unit variance within the batch β stabilizes gradient magnitudes across batches β without whitening, batches with uniformly high or low rewards can produce outsized gradients.
- Distribution of advantages within each batch β during learning, you normalize the advantages so they form a unit Gaussian β gradients become much more predictable.
- TD residual β how different the actual reward from the environment was from the value prediction β
1-step advantage estimate: low variance with potentially high bias βk-step: more variance, less bias β atk β infwe recover the full Monte Carlo advantage, no bias, highest variance.
- GAE: exponential weighting β uses an exponentially weighted average across all
k-step estimates.
- PPO-RLHF β full policy objective β PPO-RLHF combines the clipped PPO objective with a KL regularizer β two layers of regularization: 1) clipping (limits how far the policy moves per batch), 2) KL (limits total drift from the ref policy across training).
- PPO training loop:
1. Generate β sample prompts
β generate completions with current policy
2. Score
β compute rewards with reward model + KL penalty
3. Estimate advantages
β compute GAE using learned value function
4. Update (K epochs)
β clipped policy gradient + value function loss on the same batch (k=2-4 gradient steps per batch before re-generating)
5. Repeat
β new batch and sync policies
- PPO β Four models in memory (policy, value, reference, RM) β memory-intensive, which makes simpler alternatives such as GRPO attractive:
1. Policy β generates completions β update
2. Value function β estimates per-token expected return β update
3. Reference policy β KL penalty anchor β frozen
4. Reward model β scores completions β frozen
GRPO + modern variants
- GRPO β PPO for the reasoning/RLVR era β introduced in DeepSeekMath for math reasoning and became the go-to algorithm for RL on LMs β it keeps PPOβs clipped objective but drops the value function.
- Core idea β generate
Gcompletions per prompt, use the groupβs reward statistics as the baseline β no learned critic needed β simpler to implement and debug than PPO β naturally fit for RLVR as you have a clear verification and higher-variance rewards β popularized with DeepSeek R1.
- GRPO advantage β normalization of the rewards by the mean and standard deviation of the group rewards β z-score normalization: positive advantage for above-average completions β each token in completion
igets the same advantage (sequence-level, not per-token).
- Why choose token-level rewards over sequence-level rewards:
* Reward granularity
β final-answer reward vs. step/token rewards
* Loss granularity
β average by sequence vs. average by active tokens; DAPO uses token-level policy-gradient loss
* Why token-level can help:
β Local signals β KL, format rewards, PRMs, partial credit
β Long-CoT β more gradient on the tokens actually sampled
β Batching β normalize by active tokens, not examples
β DAPO β token-level loss + clip-higher + dynamic sampling for reasoning RL
- GRPO objective:
β clipped ratio like PPO + group-normalized advantages + KL penalty directly in loss
β where the clipping applies per-token, and the action in i is shared across all tokens in completion
- GRPO is PPO minus the value function, with a statistical baseline instead β without standard normalization, the GRPO-style advantage estimate becomes equivalent to RLOO up to a scaling constant
GSPO β sequence-level ratios (objective clipping)
- Aggregating per-token importance ratios across long sequences is numerically unstable β a single token with a large ratio can dominate the update β GSPO uses a geometric mean, a single length-normalized importance weight per response β the full objective mirrors GRPO but with a sequence-level ratio β the clipping range now operates on a per-token average scale, making it comparable across different completion lengths.
- Introduced by Qwen to change the ratio in GRPO to make it sequence-level rather than token-level (adds a bit of buffer before clipping) β helps to handle instability when training large models like MoE, since you can have specific tokens that can have way bigger variations in probability because of the routing of the experts and how that can fluctuate.
- On token-level vs. sequence-level rewards β if the reward is only final-answer correctness, sequence-level advantage is still natural.
CISPO β clipping only on the importance sampling (weight clipping)
- CISPO clips the importance weights themselves rather than the objective, using a stop-gradient.
- Key difference from PPO β every token still receives a gradient signal, the weight just bounds how much itβs amplified β asymmetric bounds allow more aggressive reward-increasing updates (introduced in the MiniMax M1 paper).
πΆ Lecture 4 β‘οΈ Implementing RL Algorithms for LLMs
Translating the policy gradient into code
- From the objective and its gradient, the gradient says β for each token, compute the direction that makes it more likely, then scale it by how good it was (
psi_t).
- All methods (REINFORCE, RLOO, PPO, GRPO) optimize the same core objective; they differ in
psi_tand how updates are bounded.
PPO
- Multiple epochs and minibatches β PPO (and GRPO) optionally reuse each rollout batch for multiple gradient steps β clipping activates whenever the new policy has drifted from the old β two mechanisms cause this:
1. Minibatching
β split the rollout batch into smaller minibatches to allow a larger total batch size to fit on a certain GPU setup
β after updating on the first minibatch, the policy has changed
β so later minibatches in the same epoch already see rho_t different from 1
β clipping can activate even in k=1 epoch
β with k=1 and no minibatching, the policies are the same, ratios are always 1
β clipping never activates, and PPO reduces to vanilla policy gradient with GAE
2. Multiple epochs
β loop over the full batch k times to learn more from a given rollout (which can be expensive)
β each pass sees a more-updated policy, making ratios drift further
β typical k=2-4
β after ~6, the policy is too far off-policy
- Batch size matters more in RL β gradient noise scale is orders of magnitude higher because gradients come from Monte Carlo rollouts, not labeled data:
* OpenAI data 2
β critical batch size was in millions of transitions
* PPO specifically is not batch-size invariant
β clipping couples batch size to effective step size, so you canβt compensate with learning rate
* PPO plateaus
β often caused by noisy loss estimates, which can be resolved with more samples
* Large batches reduce gradient variance proportional to 1/N
β in RLHF, this is one of the cheapest ways to stabilize training (more effective than most hyperparameter tuning)
GRPO
- Training infrastructure:
* Theory (on-policy): generate β update β generate β update
* Each batch of completions is scored and used for a short update window (one or a few epochs), then discarded
* Reality (async)
β generation and training overlap on different GPU groups for better throughput
β the model used for generation may be 1-N steps behind the training model
* Trade-off β perfect on-policy is slow β slight staleness is usually fine
- Modern RL for LLMs splits compute into two groups:
* Actors β inference GPUs β generate completions using vLLM or similar
* Learners β training GPUs β compute policy gradient updates
* A process management library coordinates data flow between them
* Modern weights are synced periodically from learner β actor
Bandit-style vs MDP-style RLHF
- Bandit-style:
* One reward per completion
* Sequence-level psi_t broadcast across tokens
* Used by default in REINFORCE, RLOO, GRPO
- MDP-style:
* Each token is treated as an action
* Per-token values or advantages
* Used by default in PPO with GAE
- Most RLHF is mixed in practice β sequence-level rewards, but token-level log-probability gradients β PPO-style RLHF usually starts from a sequence reward model score, then gets token-level via KL shaping and GAE.
- What to monitor during training:
* Reward/mean β should be steadily upward, no spikes/oscillations/plateaus
* KL/mean β gradual increase, no explosions or flatness
* Loss/policy β decreasing, no divergence or NaNs
* Metrics/clip_frac β 5-30%, not 0 or >50%
* Generation/length β stable or slight increase, no monotonic increase
* Policy entropy β slow decrease, no crashes to 0
* Also monitor: eval scores on held-out benchmarks, and read sample outputs for coherence
πΆ Lecture 5 β‘οΈ The Rise of Reasoning Models
Why does RL work now?
- Stability is much more tractable β still a first-class research problem (entropy collapse, long-horizon credit), but tooling and recipes are mature.
- Base models are good enough β multiple sources suggest RL reasoning training became viable with models ~2024 onwards.
- Verifiable domains provide clear signals β math and code give unambiguous rewards, avoiding reward hacking problems of RLHF.
Timeline
* DeepSeek R1 (Jan, 2025)
β Cold-start SFT
β large-scale RL
β distillation of smaller models
* Kimi 1.5 (Jan, 2025)
β PPO/GRPO-style RL on Chinese and English reasoning data
β difficulty scheduling and online filtering to keep gradients useful
β progressive length extension to reduce overthinking while enabling long CoT
* Open-reasoner-zero (Feb, 2025)
β minimalist replication
β fully open
β training code, curated RL data, model weights
β vanilla PPO with GAE and simple rule-based rewards
β no KL penalty
* Qwen (April, 2025)
β hybrid reasoners
β new open-weight standard
β toggleable thinking /think and /no_think modes
β mirror the R1-style multi-stage pipeline
β lightweight models use strong-to-weak distillation
β off-policy outputs, then on-policy teacher-logit-matching
* MiMo (April, 2025)
β end-to-end reasoning pipeline (from pre- to post-training)
β three-stage data mixing during pre-training - 25T tokens)
β multi-token prediction during pre-training
β multi-domain RL to prevent over-optimization on a single task type
β clear intermediate checkpoints within post-training
* OpenThoughts 3 (June, 2025)
β SFT data recipes (primarily an open reasoning data release)
β open reasoning traces
* MiniMax-M1 (June, 2025)
β CISPO clips importance-sampling weights instead of dropping high-update tokens
β introduces FP32 LM head inference-training mismatch plot
* GLM-4.5 (July, 2025)
β reasoning broadens into agentic work
β expert-model iteration plus RL for agent, reasoning, and general chat skills
* Olmo 3 Think (2025)
β fully open reasoning model
β releases: stages, checkpoints, data, infrastructure, hyperparameters
β interesting DPO findings
* DeepSeek V3.2 (December, 2025)
β pushes R1 recipe into tool use and agent environments
β open-weight MoE successor to V3.2-exp
β integrates thinking directly into tool use, no longer distinct reasoning models
* Nemotron 3 Nano (December, 2025)
β efficient agentic reasoning
β hybrid Mamba-Transformer MoE
β post-trained with SFT, multi-environment RLVR, and RLHF
* MiMo-V2-Flash (Dec, 2025)
β multi-teacher on-policy distillation (MOPD)
β train a specialist teacher per domain
β then distill them into a single student that learns from dense, token-level rewards
β not just sequence-level outcome rewards
* Nemotron 3 Super (March, 2026)
β 120B/12B MoE, LatentMoE, MTP, native NVFP4 pre-training (quantization)
β fully open: checkpoints, training data, recipes
* Arcee Trinity-Large-Thinking (April, 2026)
β frontier reasoning model tuned for long-horizon multi-turn tool use
β Apache 2.0 weights, SFT + RL pipeline
* DeepSeek V4 (April, 2026)
* Gemma 4
Recipe decisions
- Offline difficulty filtering β pre-sample
Ncompletions/prompts, keep prompts at~20%pass rate. - Online filtering and curriculum β skip prompts now too easy/hard, save harder problems for later.
- Zero-gradient filtering + active sampling β drop groups where all
Gcompletions pass or fail, then refill the batch. - KL penalty β dropped, now coming back β single-turn math RLVR usually sets
beta = 0β long-horizon, off-policy agentic training is reviving a KL/reference term for stability. - CISPO β clip importance-sampling weights instead of making high-update tokens.
- Normalization choices β how you average advantages and loss quietly changes the objective β normalize advantages across the whole batch, not per prompt group, so a few easy/hard prompts donβt dominate.
- Format rewards β reward valid
<think>blocks and extractable answers; usability, not correctness. - Language consistency rewards β penalize language switching inside a trace.
- Length control β progressive length extension β the maximum length of training, which rewards concise thinking, small penalty, overlong filtering for throughput.
- Findings - text-only RL boosts multimodal and mid-training sets the RL ceiling.
πΆ Lecture 6 β‘οΈ Direct Preference Optimization (DPO)
πΎ Your language model is secretly a reward model.
DPO (Rafailov et al., 2023) was a breakthrough in the accessibility of RLHF research β when the paper was released, many groups were struggling to open-source replications of RLHF pipelines (most aligned models were just SFT) β Zephyr-Beta, Tulu 2, etc. reiterated the post-training research.
- The promise of DPO:
* No separate reward model
* No reinforcement learning loop
* Just a single, directly differentiable loss on preference pairs
- Classic RLHF is three moving parts:
1. Collect human preference pairs
2. Train a reward model
3. Optimize the policy against the RM with RL (e.g., PPO), under a KL penalty
4. DPO collapses 2 and 3 into one supervised-style loss
β the optimal RLHF policy and the reward model are two views of the same object
Deriving the Optimal Policy
1. Start with the RLHF optimization problem
β we want to find the policy that solves the equation without RL
2. Fold the KL into the expectation
3. Flip to a minimization
4. Fold the optimization target into one log-ratio
β introduce the reference reweighted by reward
β multiplication of each responseβs probability by an exponential factor (on beta) larger for higher rewards
β (i.e., setting how aggressive reweighting is)
5. Introduce the partition function Z(x) and βenergyβ minimization
6. Turn the objective into a KL divergence
7. Apply Gibbsβs inequality to get the optimal policy
β β¨ DPO is the gradient on it β¨
β the reward model never has to be built, but itβs hiding inside the policy!
DPO weaknesses, implementation
- A subtle risk: the chosen probability can fall β The DPO loss only cares about the margin between the chosen and rejected log-ratios, not the absolute values β the model can lower the loss by pushing the rejected probability down faster than the chosen, even while the chosen probability also falls β a reason some practitioners add an SFT term on the chosen response, or fixes like Cal-DPO.
- The beta parameter sets the strength of the KL constraint relative to reward maximization β large betas, the policy stays close the reference policy; small betas, the policy is free to deviate and can over-optimize.
- DPOβs final KL distance is static β it steps directly to the optimal solution implied by the dataset and the chosen beta. β online RL instead takes steps based on freshly sampled batches and a per-sample KL penalty β some RL runs even include dynamically adjusted KL controllers.
- DPO vs. RL/PPO (offline vs. online) β 1) train on a fixed dataset collected ahead of time vs. generate fresh completions during training, score with a reward model; 2) simpler, stable, faster vs. more compute and complexity, 4 models in memory; 3) limited by the coverage of the dataset vs. can explore new regions.
- Models today: Olmo 3, Nemotron 3 (mixed preference optimization: DPO + binary classifier optimization in one offline stage, over data scored by a generative reward model), Liquid AI LFM2.
πΆ Lecture 7 β‘οΈ On-Policy Distillation & Using Synthetic Data
- Synthetic data:
* Lowered the price of post-training experiments; the time-to-collect was faster, enabling RSI arguments
* Synthetic data in post-training only worked once GPT-4-class models arrived
β Llama 2 and GPT-3.5-Turbo were not reliable enough to generate or supervise data
β LLM-as-a-judge ability emerged in the GPT-3.5 to 4.0 jump
* Today, leading models need synthetic data to reach the frontier
β distillation is the general word to transfer capabilities
The Roles of Synthetic Data
- Spans the entire pipeline:
* writing prompts from seeds
* modifying / expanding existing prompts
* generating completions to prompts
* Providing AI feedback to create preference data
* Filtering completions for quality, labeling preferences
* verifying answers as rewards for RL
- Model collapse, βas diversity drops, rare facts and styles are underrepresented and small mistakes compound across iterationsβ β not a worry anymore β collapse can be avoided by 1) mixing real/human data, 2) using diverse teachers, 3) deduplication, 4) strong quality filters.
The path to on-policy Distillation (OPD)
- Knowledge distillation (Hinton et al., 2015) uses soft labels (the full distribution over next tokens) rather than the one-shot target of next-token prediction β to apply it to autoregressive LMs, decompose the loss per token.
- Notation β
sis the source prompt,uis the teacher trajectory,Vfor tokenizer,qfor the teacherβs next-token distribution andpfor students,afor the student-sampled completion in the on-policy / RL notation.
- Word-level (per-token) distillation β standard teacher-student distillation for an LLM β expensive but tractable (
|V|probabilities per position) β matching over whole sequences is the hard part.
- Exposure bias β the propensity for the student to accumulate errors β offline KD samples teacher trajectories
u ~ pi_Tand matches per token (q β pi_T, p = pi_theta) β at test time, these policies are different (different distributions) β DAgger analogy (compounding error).
- From off-policy to on-policy β sampling from the student (rather than the teacher) minimizes a lot of the distributional errors:
* In offline KD, a single suboptimal token can nudge the student generation slightly out-of-distribution
β the model is more likely to err again (as it has never seen the token in training)
* On-policy distillation iteratively samples from the student and supervises it with the teacher on its own visited states
* MiniLLM introduced a reverse-KL objective inside a policy-gradient frame
β concurrent work connected on-policy KD to imitation learning (closer to the modern distillation form used today)
Forward vs. Reverse KL
- Sampling completions from the student is what puts
pi_thetaon the left of the KL - which flips its direction (estimating the KL and its direction relies on which distribution you sample from).
- Offline KD / SFT (
~ pi_T) β the expectation is over the teacher (off-policy, a fixed teacher dataset) β mass-covering (weighted by teacher mass; wherever the teacher has mass andpi_theta β 0, the log-ratio blows up and the student must cover everything the teacher might say).
- Online distillation (
~pi_theta) β the expectation is over the student (on-policy, you sample the model you are training; mode-seeking - weighted by the studentβs own mass, penalized only where it puts probability the teacher dislikes, so it collapses onto the teacherβs modes).
- KD distance as an RL advantage (i.e., directly as a reward) β substitute the negative per-token reverse-KL contribution as the advantage ( thinking machine post ) β for a sampled token
a_tat states_tit becomes just a log difference, not the policies on them:
* Tokens more likely for the teacher β positive advantage; less likely β negative
* The teacher log-prob gap is dense, token-level feedback
β potentially richer than a sparse verifiable reward or a single scalar reward-model score
* This layers into modern RL machinery
β e.g., add it alongside GRPOβs group-level normalization for more complex reward shaping
- Multi-teacher on-policy distillation (MOPD) β use several teachers - domain specialists or earlier checkpoints β each with a pre-prompt mixture weight β at scale: organizations divide labor, train expert teachers that later distill into one final student β DeepSeek-V4-Pro, MiMo-V2-Flash.
- On-Policy Self-distillation (OPSD) β pushing the frontier β the teacher is the same model conditioned on privileged information β gradients will teach the model the tokens after the hint were a mistake, absorbing the lesson with an OPD-style loss β Cursor Compose 2.5, Kimi K2.5.
AI Feedback and Constitutional AI
- RLAIF (RL from AI feedback) β using AIs to approximate the human-data step, starting with pairwise preferences β cost advantages.
- Building specialized judge models:
* Research on how LLMs are inconsistent evaluators and show self-preference bias (they favor their own generations)
* Dedicated judge / critic models exist (not widely adopted in documented post-training recipes)
* Equilibrium β frontier models are already trained hard for judging, so you rarely need your own
β unless your task has private data
- Constitutional AI (CAI) β Anthropicβs post-training method for Claude models β the earliest documented, large-scale use of synthetic data for RLHF β generates synthetic data in two ways, one for instructions and one for preferences:
* It builds an SFT pipeline, rewriting the prompt based on the sampled principle
* Then constructs preferences by giving a feedback model
β the model selects which answer is higher quality and more aligned with the principle
Rubrics: Prompt-specific AI Feedback
- A popular tool for scaling RL on the long tail of domains β used to help with domain-specific evaluations and any other place domain expertise needs to be trained into the models
- Emerged in late 2024 β 2025 as LLM judges and synthetic-data practices matured β likely a function of making RL more broadly accessible to frontier post-training
- Per prompt β quality and robustness to over-optimization
πΆ Lecture 8 β‘οΈ Preference Data
- Logic to numbers; choices scored:
* Port Royal Logic (1662)
β decision quality = outcome weighted by its probability
* Benthamβs hedonic calculus (~1800s)
β weigh all of life on one complicated, but common scale
* Ramsey, Truth and Probability (1931)
β first to quantify preference and belief together as the way that individuals make probabilistic decisions
πΎ To judge what one must do to obtain a good or avoid an evil, itβs necessary to consider not only the good and the evil in itself, but also the probability that it happens or not.β - The Port Royal Logic, 1662
- Von Neumann-Morgenstern Utility theory (1947) β if your preferences obey a few axioms (completeness, transitivity, continuity, independence), they can be represented by a single utility function and a rational choice β maximizing expected utility. β RLHF fitting a scalar reward.
- Preferences drift during and after labeling; they are context- and framing-dependent, can be intransitive, and highly multidimensional.
- Where utility theory breaks down:
* Arrowβs impossibility theorem (1950)
β no voting rule aggregates individual preferences into a collective one while satisfying a few basic fairness criteria
* Sen, Behaviour, Choice, and Values (1973)
β choice is not the same as preference
β revealed-preference theory is too thin
* Hirschman, Against Parsimony (1984)
β people have preferences over their preferences, so preference may be unmeasurable
- The other root β optimal control and RL β a machinery for optimizing a reward β these guarantees assume a single closed-form reward:
* Bellman (1957)
β MDPs and dynamic programming β origins of optimal control / control theory
* Sutton (1988)
β temporal-difference learning for credit assignment (instead of learning from lots of past data)
* Watkins (1992)
β Q-learning (learning value)
* DQN (2013)
β deep RL at scale, AlphaGo / AlphaZero (2017) β mastery from self-play
- Rewards are a scalar signal to maximize, while preferences are unstable and not scalar β reducing preferences to rewards made the optimization format tractable but itβs the root cause of unsolvable biases in RLHF and preference data.
- A learned reward model is a moving, noisy proxy, not a ground-truth reward β inverse RL: learning a reward from behavior (e.g., on-policy distillation) is conceptually close but absent from RLHF practice β we inherit RLβs optimizers without its guarantees.
Preference data β Trade-offs in Practice
- Ratings β a score on one completion in isolation; good ad metadata.
- Rankings β relative comparisons, often on a Likert scale, early Claude used an
8-point scale; UltraFeedback pairs high- vs. low-rated completions. β In practice, almost everyone trains on pairwise rankings, binarized to chosen/rejected for the Bradley-Terry loss - and keeps ratings on the side.
- Likert scale β the famous scale that records a preference as an ordered, graded judgment - not just which answer wins, but by how much, on a symmetric scale with an optional neutral middle. β ordered judgment.
- Beyond pairwise preferences β richer signal, harder collection:
* Directional / single-bit labels
β trained with Kahneman-Tversky Optimization (KTO)
* Token-level / fine-grained feedback
β label specific tokens, or token-spans as good or bad
* Natural-language feedback
β written critiques instead of a label
- "The nature of preferences": Preference evals β Arena; Spec β data β behavior.
πΆ Lecture 9 β‘οΈ Over-Optimization and RLHFβs Bad Reputation
- OpenAIβs sycophancy problem, April 2025 β what went wrong:
- The model update had a reward signal from user feedback via an RM for RL data from ChatGPT
- Under RL, that signal overpowered the primary rewards
β optimize the easiest objective to move
- Not caught by evals
β offline benchmarks looked good and A/B testers preferred the model
β no deployment eval tracking for sycophancy
- Reward-model accuracy is a proxy for a proxy (a learned model, from data that incompletely captures a complex distribution - proxy for truth, as over-optimization is common).
- Critique of RLHF being βstyle transferβ β the Superficial Alignment Hypothesis (Zhou et al. 2023) is the strong version: knowledge is learned in pretraining; alignment just picks a format and tone.
Over-optimization
βAny observed statistical regularity will tend to collapse once pressure is placed upon it for control purposesβ - Goodhart, 1984
- RL is a very strong optimizer β it pulls all the available reward out of the environment. β in RLHF, the reward is a learned model, at best correlated with downstream quality. β optimizing the proxy makes the true objective better in early training - then worse.
- Over-optimization vs. overfitting β in overfitting, the model memorizes training examples rather than generalizing patterns (so training accuracy improves while accuracy degrades) β in over-optimization, the model genuinely improves at the proxy objective: the reward modelβs scores (including on validation set), but that objective diverges from the true goal - itβs an imperfect measurement problem.
- Anthropicβs version of over-optimization β split preference in half and train two
52Bpreference models (test and train) β training with RL on the training set only, inducing a KL distance from the initial policy.
π€£ βI cannot provide instructions or advice on how to harm or destroy Linux processes or any other living being. Itβs not ethical, legal, or morally rightβ¦β
- Source of errors: 1) approximation (the RM cannot perfectly fit preferences); 2) estimation (the RM overfits its training set); 3) optimization (the policy trains too hard against it).
- Potential mitigations: 1) bigger policies (more ways to gain reward at small optimization distances); 2) reward-model ensembles; 3) changed optimizers; 4) direct alignment over-optimizes but makes the trade-off easier to pin; 5) best-of-N sampling spends far less KL than online RL β the main lever is KL penalty (and more careful data).
- Over-optimization enables misalignment. β reward hacking in scaled-up RL on verifiable and agentic tasks β models exploiting graders, test harnesses, and tools rather than solving the task.
- Preference tuning reliably boosts LLM-as-a-judge chat evals (AlpacaEval, MT-Bench) - gains that do not transfer proportionally to Arena or real usage. β DPO leads to improvements in human preference evaluation but degradation in benchmark evaluation.
- Preference tuning method from βDirect Nash Optimizationβ (April, 2024): reports a
7Bmodel beating GPT-4 on AlpacaEval β not quite true, hacking some measurement β a lot came from length expansion, where longer generations were judged as better.
- The honest retrospective β RLHF earned its bad reputation on style failures, but the same tools, used carefully, are now central to modern post-training.
πΆ Lecture 10 β‘οΈ Regularization in RL, Why RL Generalizes, and Why SFT Forgets
How do these optimizers change the distribution of the models? How do we control it?
- The RLHF process is the RL step maximizing reward from the RM minus a penalty from drifting from the reference model β what happens with this penalty?
- RLVR has regularization too, but different best practices β same RL loop, different reward source β a verification function instead of a reward model β reasoning models often dropped the KL penalty to enhance learning β with large-scale tool use, regularization is coming back, aimed at drift from the sampling distribution, not a KL penalty to a reference model.
Explicit KL Penalty
- KL control for RL predates LLMs β dialogue agents, then fine-tuning pretrained models. β this penalty is a reverse KL: estimated by sampling from the policy you are training and scoring against the reference model β it punishes the policy for putting mass where the reference would not.
- Intuition for KL is that itβs an (optimization) distance, but when implementing itβs not a distance/true metric as itβs not symmetric (commutative).
- Measuring KL in practice β sampling from
Pturns the definition into an expectation β practitioners watch the KL curve during training (a very large KL usually means a bug or a broken model).
- Static or dynamic KL penalties? Beta began as a feedback controller β the first RLHF for LMs paper did not fix beta; it picked a target KL and let the controller chase it. β Before RLHF, PPOβs original adaptive KL penalty variant doubled or halved beta around a target, and constrained RL later made the control framing explicit with full PID controllers on the penalty multiplier. β modern practice is back to small static beta, and in many RLVR setups, no KL at all.
RL Optimization is a Reverse KL Minimization
- Whatβs the shape of RL optimization and how it relates to KL? RL samples from itself β with the penalty on, maximizing the objective is exactly a reverse-KL minimization toward a reward-tilted reference β this is only true with the KL penalty in the optimization β but on-policy sampling still biases RL toward KL-minimal solutions.
- The optimized policy as the reward-tilted reference β the βtiltβ shifts probability mass toward high-reward completions while staying inside the referenceβs support - large beta tilts barely at all, small beta concentrates on the highest-reward completions. β with the penalty included, RL doesnβt just use a reverse KL, the whole objective is one, pointed at the reward-tilted reference policy.
- Minimizing the SFT loss is minimizing forward KL (to the data) distribution
- Forward KL (SFT, standard KL) vs. Reverse KL (RL) β samples from the target (fixed database, mass-covering, wherever the target has mass and
pi_thetagoes to zero, the loss blows up, the model must spread to cover everything) vs. samples from the policy itself (mode-seeking, only penalized where it places mass, so it concentrates on high-reward modes).
Why RL Generalizes More
- SFT Memorizes, RL Generalizes (Chu et al., 2025) β RL-based post-training carries implicit regularization from its on-policy structure alone.
- Naive read β forward KL is mass-covering, so SFT should preserve every mode, while mode-seeking RL should collapse onto one and forget the rest. β this assumes a unimodal policy, and LLMs are multimodal.
- RLβs razor (Shenfeld et al., 2026) β lower KL drift for equivalent performance β βAmong the many high-reward solutions for a new task, on-policy methods such as RL are inherently biased toward solutions that remain closer to the original policy in KL divergence.β
Other Tools for Control Optimization
- Pretraining next-token prediction gradients; NLL alongside DPO; margin loss for reward models.
- A trust region on the sampling distribution β DPPO (Divergence Proximal Policy Optimization) masks tokens by a directly estimated divergence between the rollout and training policies, instead of PPOβs per-token ratio clip.
- The per-token ratio is a noisy one-sample estimate of that divergence. β this is not the reference-anchored reverse KL but a drift control against the sampling distribution, a trust region on each update rather than a penalty in the objective.
- Even with no penalty, on-policy RL is implicitly regularized - SFT memorizes, RL generalizes.
πΆ Lecture 11 β‘οΈ How Language Models Use Tools and the Path to Agents
What are the limitations of Model weights alone?
- What an LLM is has changed β an LLM today is: model weights (the trained network: foundation of knowledge, reasoning, style) + tools (the actions the model can request: search, code execution, file edits, APIs) + harness (the software loop around the weights that executes those requests and feeds the results back into the context) β now systems, instead of static weights.
- The hardest evals today are e2e tasks in complex containers!
A short history of models using tools:
* 2015-2020, precursors
β neural programmer-interpreters execute programs with neural networks
β retrieval augmentation pulls in outside knowledge
* 2011
β WebGPT browses the web, trained with human feedback
* 2022
β TALM bootstraps tool-augmented training data
β PAL offloads computation to Python
β ReAct interleaves reasoning and actions
* 2023
β Toolformer teaches itself APIs with synthetic data
β Gorilla scales to 1,645 APIs, ToolLLM to 16k+
β OpenAI ships function calling in the API and code interpreter in ChatGP
* 2024
β MCP
* 2025
β o3 makes multistep tool calls inside its reasoning
* 2026
β terminal and coding agents become the frontier of post-training
- ReAct: Reasoning and acting are one generation β before, reasoning (CoT) and acting (tool calls) were separate literatures β interleaving them in one token stream is the new pattern for agents.
- Toolformer: Models teach themselves tools β via a self-labeling/synthetic data mechanism β 1) Prompt the model to insert candidate API calls into its own pretraining text, 2) Execute the calls, 3) Keep only the calls whose results reduce perplexity on the following text, 4) Fine-tune on the filtered corpus.
- How tool use is evaluated β 1) Schema-level (JSON validity, etc.); 2) Breadth (ToolLLM/ToolBench spanning 16k real-world APIs); 3) Reliability (e.g., tau-bench measures
pass^k, succeeding on allktrials); 4) e2e (terminal-bench runs agents on real tasks in containers with verification tests; long-horizon benchmarks).
Infra - How tool calls actually work
- The model only sees tokens β tools live in the system prompt β training data for function calling looks like ordinary post-training data, plus a system prompt declaring the available tools as JSON schemas. β the model learns to emit calls matching the declared schemas and expect results in context.
Training for Tool Use
- Same post-training applies:
* SFT on tool trajectories β formatting and tool selection; establishes the skill
* Preference tuning still works, but is less popular
* RL with environment feedback, where the action is in frontier model training
- For RL, environments are the bottleneck β scaling environments means synthesizing them (difficult control, personas, verifier diversification) β environments have a ton of new infra + complexity failure modes (API keys becoming inactive, CPU resource limits, bandwidth limits, missing data the model was told it has, etc.).
- TMax β an open recipe for terminal agents β RL over TMax-
15Kenvironments with outcome-only rewards. β DPPO, a GRPO variant, designed to help with agentic stability. β TMax-9B: 27.2% on Terminal Bench 2.0 (the strongest open-weight model under10B, beating the32Bvariants of prior work).
- Most of the effort goes into stability during scaling, rather than speed or learning efficiency β without intervention, runs were often unstable, collapsing after
300training steps β train/inference numerical mismatch β the inference engine and the trainer disagree on logprobs (fixes: anFP32LM head, and DPPOβs masking of tokens where the two diverge) β a standard run: H100 nodes (2 for training, 6 for inference, 2-3 days).
- Kimi K3 (July, 2026) β lots on environments and sandboxes (
51,219,741sandboxes created during training, microVMs that checkpoint in 133ms, and living environments, where one rollout can span thousands of tool calls β sandboxes sit idle up to98%of their lifetime waiting on inference).
- Open models train across multiple harnesses so deployment is smooth β Kimi K3 says that training with a single fixed agent harness can cause a model to overfit β Nemotron 3 Ultra trains every task under at least two harnesses.
- Nemotron 3 Ultra β as environments multiply, βeach domain contributes only a relatively small number of samples to any given training batch, diluting the per-domain learning signal.β β a common answer is to train domain experts with RL, then merge them via multi-teacher on-policy distillation (MOPD).
- Reward hacking β GLM-5βs slides-RL policy discovered
overflow: hiddento make an overflowing slide measure16:9, and flex-padding to stretch short ones (the fix was patching the renderer, not the reward). β Nemotron deletes future git commits and firewalls GitHub so SWE agents canβt read the gold patch β Kimi K3 ships a kernel-exploit detector and public/hidden verifier pairs.
- More headaches:
* Long-tail rollouts
β one trajectory makes 2 calls, another makes 200
β async is designed to help, but long-tail is challenging
* Credit assignment β one sparse reward over a 10^5-10^6 token trajectory
β expensive rollouts also make GRPO-style algorithms more costly
* Verifier gaming
β TMax rollouts were caught replacing test files with no-ops and faking binaries with simulated logs
* Harness-native training
β production agents live inside complex deployment harnesses (which may not be in training data)
πΆ Lecture 12 β‘οΈ How Evaluation Has Evolved with Frontier Model Progress
- Frontier evaluation is harder than it has ever been β benchmark-based release reactions barely matter at the frontier.
- The task length frontier models can complete doubles every
~7months.
The Eras of Post-Training Evaluation
- Benchmarks mirror the training goals of their era β the key to understanding evals: popular benchmarks are a reflection of the training best practices of their era:
* Chat era (2022-23) β basic knowledge and chat style
* Multi-skill era (2023-24) β post-training improves more skills than just chat
* Reasoning & tool eras (2024-26) β long chains of thought
* Agents & Real work (now) β e2e knowledge-work tasks inside products and harnesses
- Base models (before post-training):
β few-shot prompting β base models canβt take a bare or formatted question
β eval prompts carried example software of the patterns so the model continues the pattern
β canonical evals: 5-shot MMLU, 8-shot GSM8K
- Multiple ways to extract an answer from a model:
* log-likelihood scoring:
β compare the probability the model assigns each answer option
β either just the letter A or the full answer string
β no sampling, fully deterministic
* Generation + exact match
β sample a completion, extract the answer
β mirrors real usage, and is standard for post-training since ~2024
β aggregating multiple completions/samples gives majority voting
* Generation and extraction gave rise to answer-extraction formatting bugs
β which only became more complex with agentic models today
pass@kβ the probability that at least one ofksamples solves the problem β the naive route generates exactlykand reports whether any passed; this is a high-variance coin flip per problem, and plugging a small-sample success rate into1 - (1 - p)^kis biased β the fix, from the Codex paper: samplen >= kcompletions, count thecthat pass, and average an unbiased estimator over problems β the knobs interact: higher temperature can hurtpass@1but helppass@100, so a reportedpass@1depends onnand the sampling settings, not just the model.
- Chain of Thought emerged to enable progress on harder problems β few-shot examples that show intermediate steps let models reason before answering β when people were still prompting base models, adding CoT made math and reasoning scores jump.
- Entering the chat era β zero-shot instruction following β instruction tuning and then RLHF changed the way people expected to use models β the models learned to directly answer questions:
* LLM-as-a-judge emerged as questions became open-ended (and evals imitated RLHF training)
* Canonical evals β MT-Bench, AlpacaEval, and the community-scale Chatbot Arena
* MCQ evals like MMLU stayed in the mix - now answered zero-shot, sampling the answer letter at temperature 0
- The emergence of zero-shot prompting took time β when training Tulu 3 (summer 2024), many evaluations were a mix of zero-shot and few-shot prompting. β knowledge: MMLU, PopQA, TruthfulQA β Reasoning: BigBenchHard, DROP β Math & Code: MATH, GSM9K, HumanEval β Instruction following: IFEval.
- Modern eval suites carry per-benchmark prompts tuned for formatting β sampling settings joined the prompt as part of the eval: reasoning models need temperature > 0 for their best scores.
Can we trust the number?
- Evaluation variance is everywhere:
* Sampling at temperature > 0 means re-running the same eval on the same models moves the score
* During Olmo3, we measured it β std. dev. across 3 runs of 14 models, per benchmark
* Most reasoning-era evals sit between 0.25 and 1.5 points of noise - before anyone changes a prompt or a sampling setting
- Managing eval noise:
* avg@k is the rescue
β LiveCodeBench was noisy and cheap β retuning 10x moved it from high-variance to very stable
* Variance also leaks in from infrastructure
β batch size, tensor-parallel settings, numerics of long generations
* Practical rule
β a ~1-point gap between two press releases is noise
- What evals are actually for inside frontier labs:
* Lab hillclimb on ~50 prioritized evals and report the public suite (subset) at the end
* The real product of a good internal evaluation is statistical power
β less noise on the signals used to compare training runs
* Sometimes the βtest setβ is just good data
β MATH and GSM8K train splits are high-quality and crucial at a time
β if a lab doesnβt track that evaluation, training on them is a rational choice
* Human A/B testing and Elo stay in the loop for what benchmark makers canβt measure
- Established evaluation harnesses:
* Inspect
* lm-evaluation-harness
* LightEval
* OLMES
* HELM
* Eval Gauntlet
* olmo-eval
* verifiers from Prime Intellect
πΆ Lecture 13 β‘οΈ An Introduction to Character Training
- Changing how a model behaves (in order of increasing effect and effort):
1. Prompt it
2. Steer its activations (manipulate the modelβs internal state with no gradient updates)
3. Train it (character training, post-training designed to craft traits and manner into the weights)
Character Training
- Features are sequences of words/tokens it repeats; behaviors are how those link together.
- Pipelines that control the specific language in training data.
- Constitution: principles that are inputs to the training process β principles sampled inside the data-generation pipeline, not a statement of final behavior; the modelβs final behavior is an emergent result of running the pipeline over them (started by Anthropic).
- Model spec: states the intended final behavior β public document of goal model behaviors to guide experimentation and decision-making, allowing users to understand if a behavior/issue was an intended action they donβt agree with or a bug in the technical process (to be fixed later) β important sign of intent when compared to a more vague constitution (OpenAI).
Who is a model spec for
- Model designers: clarity on which behaviors are wanted or not, easier prioritization decisions on data, a bigger picture above complex evaluation suites.
- Developers: a way to tell which behaviors are intentional vs. side effects of training.
- The observing public: one of the few public sources on what is prioritized in training; the substrate for regulatory oversight and effective policy.
Character Training in Practice (Maiya et al., 2025)
- Workflow:
1. hand-written trait constitutions for multiple personas
2. pairwise preference data for DPO
3. synthetic introspective data for SFT
- Evals β instruct the model to embody one of two traits without verbalizing the choice, have an LLM judge which trait each of
25kresponses expresses, and compute an Elo score per trait.
Character Elicitation Without Gradient Steps
β¨ Persona vectors β concepts are directions in latent space (Word2vec) β extract a traitβs direction from its description alone β an LLM writes prompt pairs to elicit / suppress it β steer by adding it back at inference (traits dial almost perfectly linearly and compose). β¨
- The assistant axis β where the default persona lives β extract vectors for
275+character archetypes and run PCA across them.
- Activation capping for precise intervention β same tool as persona-vector steering β add a vector to the activations at inference, but conditionally throughout 1) steering (persona vector slide), 2) capping (check how assistant-like activation is) β result: at some turn, the drifted model maintains the assistant personality.
- Persona subnetworks β mask in weight space β Frankle & Carbin (lottery-ticket-flavored hypothesis), 2019 pre-trained models already contain persona-specialized subnetworks β from a few hundred calibration examples, score each connection by weight magnitude vs. source-neuron activation and keep the top-
kper row as a binary mask β Switching personas by swapping masks over frozen weights with only the most influence on the output β the downside is costly capability regression by turning off parts of the network β not scalable (yet).
- Big open question in character training β a spec is only as good as the effort spent making the model follow it.