A guide to LLM post-training

Pre-training gives a language model general background knowledge. Post-training teaches it how to be useful for your specific task. In this guide, we break down the five main LLM post-training techniques: supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), direct preference optimization (DPO), reinforcement learning from verifiable rewards (RLVR) with Group Relative Policy Optimization (GRPO), and distillation. We'll explain how each one works, when to use it, and how they fit together in a training pipeline.

Background: how LLM pre-training works

During pre-training, the model sees enormous amounts of data and learns patterns and relationships between inputs and outputs. Specifically, it runs a forward pass to generate predictions, computes loss to determine how wrong it was, then uses backpropagation to calculate how to update the weights to improve its responses. This process is repeated until the model captures broad knowledge about language, code, the world, or whatever data it's being trained on.

The pre-training loop: training data flows through repeated cycles of forward pass, loss computation, and backpropagation.

A base model is a generalist. To train it for a specific task, you need post-training techniques like SFT.

Supervised Fine-tuning (SFT)

SFT is simply more training based on specific examples.

Imagine you want to turn your pre-trained model into a medical scribe. The pre-trained model might have never seen examples of this task: it won’t know the right format you expect or what's clinically relevant. We use SFT to train the model further on specific, task-related examples.

SFT helps make the model consistently follow instructions, sets tone/guardrails, and teaches basic tool use.

SFT Steps

To do SFT, you:

Create a dataset with prompts and expert answers.
Train the model further on this dataset: forward pass, compute loss, backpropagate.

Step 2 can be done in different ways, depending on how much of the model you'd like to update. Some techniques include adapters, prefix and prompt tuning, BitFit, QLoRA, DoRA, or freezing most of the network and training only the last few blocks. The two most common are full fine-tuning and LoRA (explained immediately below).

Full fine-tuning

Full fine-tuning updates all the parameters in the model. Every weight gets adjusted during training: this gives maximum flexibility, but it's expensive. You need enough GPU memory to hold the full model, gradients, and optimizer states.

Larger models are often split across multiple GPUs because a single GPU doesn't have enough memory to hold all three at once. Each GPU holds a slice of the model and computes gradients only for that slice. During one training step, those gradients have to be combined before the weights can update. That exchange costs time.

You also need a complete copy of the model for every task you fine-tune.

LoRA (Low-Rank Adaptation)

To do LoRA fine-tuning, first, you freeze the original weights. When you typically fine-tune, the new weights are the original weights plus an update: $W + ΔW$ . LoRA assumes the update $ΔW$ is low-rank, which means that it can be represented as the product of two smaller matrices.

So instead of updating every weight during a training step, the model updates the weights of two smaller matrices $A$ and $B$ ( $ΔW = BA$ ), known as LoRA adapters. The model trains ~1% of the parameters, which decreases memory usage (the frozen weights don’t need gradients or optimizer states).

Use cases: non-agentic tasks like classification, labeling, summarization, voice chats, and medical scribing (not long-horizon tasks). Decode (LLM output) is under 10K tokens.

Limitations of SFT: SFT can only tell the model what a good response looks like. It can't tell the model that one response is better than another, or that a response is bad. Because of this, SFT only trains the model to sound like the examples, not to give answers humans would prefer, so it can copy the style perfectly while still making mistakes.

SFT in the post-training pipeline. Pre-training produces the base model; SFT is the first post-training stage, followed by preference training and then target RL (each covered later in this guide). SFT doesn't act alone: it establishes the basic behavior that the later stages refine.

SFT with LoRA in Baseten Loops

The following code performs one supervised fine-tuning step with LoRA using Baseten Loops.

pip install baseten-loops

export BASETEN_API_KEY="your-api-key"
export LOOPS_PROJECT_ID="your-project-id"

1from baseten.loops import AdamParams, Datum, ModelInput, ServiceClient, TensorData
2BASE_MODEL = "Qwen/Qwen3.5-2B" 
3
4service_client = ServiceClient() 
5training_client = service_client.create_lora_training_client(
6                  base_model=BASE_MODEL, rank=16, ) 
7tokenizer = training_client.get_tokenizer()
8
9def build_sft_datum(tokenizer, prompt, answer):
10  p = tokenizer.encode(prompt, add_special_tokens=False)
11  a = tokenizer.encode(answer, add_special_tokens=False)
12  # Shift so logits at position i predict token i+1
13  full = p + a
14  tokens = full[:-1]
15  targets = [-100] * (len(p) - 1) + list(a)   # -100 = ignored by the loss
16  return tokens, targets
17
18tokens, targets = build_sft_datum(
19  tokenizer,
20  prompt="What is the capital of France?\nAnswer:",
21  answer=" Paris",
22)
23
24datum = Datum(
25  model_input=ModelInput.from_ints(tokens),
26  loss_fn_inputs={
27    "target_tokens": TensorData(data=targets, dtype="int64", 
28shape=[len(targets)])
29  },
30)  
31
32# One training step = forward pass + compute loss + backprop + optimizer update
33training_client.forward_backward(data=[datum]).result(timeout=600.0)
34training_client.optim_step(AdamParams(learning_rate=4e-5)).result(timeout=600.0)

The -100 values mask the prompt positions, so the loss is calculated only on the target answer tokens.

Reinforcement learning

Overview of reinforcement learning

The reinforcement learning loop. An agent takes actions to interact with the environment, which returns a new state and a reward.

Reinforcement learning is a framework where an agent learns by interacting with an environment: at each step it observes the current state, chooses an action from a finite set of options, and the environment transitions to a new state. After each action, the environment also returns a reward: a positive or negative signal telling the agent how well it's doing. In many RL tasks, this reward is zero for almost every step, and the agent only receives a non-zero reward after a long sequence of actions.

This is the classic framing, and it comes from robotics and game-playing. For example, in chess, you are the agent (the player), the state is the board position, and your actions are the legal moves available to you. The environment is everything you don't control: the rules of the game and an adversarial opponent trying to beat you.

In many RL problems, rewards are sparse and delayed: you might play 40 moves and only receive one reward signal at the very end of the game (+1 for a win, −1 for a loss, 0 for a draw). This is the credit assignment problem: based on a sparse reward signal, you need to figure out which moves were good and which were bad.

A trajectory with sparse reward. The agent moves through states taking actions, receiving r = 0 at every intermediate step and a nonzero reward only at the terminal state. This is the credit assignment problem: a single reward at the end has to explain a long sequence of actions, with no signal about which ones actually mattered.

Post-training for LLMs maps onto the same structure, where the model acts as the agent. The state is the prompt plus whatever tokens have been generated so far, an action is the next token, an episode is one complete response, and the reward is a single score for that response from a reward model or a verifier. The environment in post-training is trivial. Each token is appended to the context, and that's the new state.

RLHF (Reinforcement Learning from Human Feedback)

RLHF requires training an additional reward model that can predict human preferences. The core training algorithm that updates the model weights based on human preference is Proximal Policy Optimization (PPO). Group Relative Policy Optimization (GRPO) is a successor of PPO introduced by researchers behind the DeepSeek models.

RLHF training uses a reward model and a policy model (the LLM itself). The policy is a function that takes a state as input and returns an action.

Reward model

The reward model translates human preferences into a numerical signal: it takes in a sequence of text and outputs a single reward value predicting how much a human would like that text.

How is the reward model trained?

The model generates several answers to the same prompt.
Humans rank the answers (“answer B is better than answer A”).
These comparisons train the reward model, whose job is to look at a response and output a score predicting how much a human would like it.

PPO (Proximal Policy Optimization)

PPO maintains a separate critic (value function) that predicts the expected return for every state. The expected return is the estimated total reward from the current state until the end of the response: "given what's been generated so far, how good will the final outcome be?"

The policy's actual reward* is compared against those expected returns, and that difference (the advantage) determines the loss. Token choices that led to better-than-expected outcomes get reinforced (in similar contexts); worse get discouraged. PPO also limits how much the policy gets updated in one training iteration.

*The reward model outputs one score r per response (not per token).

The PPO training loop. A policy model generates a response scored by a frozen reward model (r) and a trained critic (v), whose difference forms the advantage used to update both models.

RLHF is used when you have human preference data and need a learned reward model. It is best when success is subjective, like determining whether an assistant was helpful across the whole conversation.

RLHF use cases

Flagship chatbots: frontier labs (OpenAI, Anthropic, Google) use RLHF because it delivers high-quality results (e.g., OpenAI turned GPT-3.5 into ChatGPT).
Safety-critical behavior: RLHF teaches models to refuse harmful requests without over-refusing harmless ones (e.g., decline "how do I synthesize a nerve agent," answer "how do pesticides work").

RLHF limitations

Expensive: lots of compute and human labeling.
Complex: 4 models running at once (policy, reward model, critic, reference).

Reinforcement Learning from Verifiable Rewards (RLVR)

RLHF is more oriented towards aligning and refining a model’s behavior based on subjective human preferences whereas RLVR focuses on automatic and objective feedback like answer correctness.

RLVR works in an environment where we can clearly define the success of an outcome as either a pass or a fail. It learns from getting signals about whether or not the task has been completed successfully. GRPO is the algorithm that turns those pass/fail signals into weight updates.

GRPO (Group Relative Policy Optimization)

GRPO drops the critic that PPO uses. For each prompt, the policy generates a group of responses, and each one gets scored: either pass/fail (e.g., 1 or 0 for a correct/incorrect answer) or a numerical score (e.g., fraction of unit tests passed). The group's average score plays the role of the expected return. Every token in an above-average response gets reinforced; every token in a below-average one gets discouraged.

Like PPO, GRPO clips how much the policy gets updated in each training iteration. Since there's no critic to train, the policy is the only trained model, which makes GRPO much cheaper in memory and compute, and a good fit for RLVR.

The GRPO training loop. For each prompt the policy generates a group of responses, and a verifier scores them all. Unlike PPO, there is no critic.

Use cases for GRPO

Autonomous coding agents (Cursor, Claude Code, Codex)
Math problem solving
Complex tool use and task completion
Long-horizon interactive LLM agents (multi-step tasks)
Reasoning modes in production assistants like ChatGPT and Claude

DPO (Direct Preference Optimization)

DPO is a way to align a model to human preferences without a reward model or reinforcement learning. As the DPO paper’s title states, "your language model is secretly a reward model."

The data is a preference dataset where each example is a prompt with two model-generated responses: one chosen (preferred by a human) and one rejected.

Two copies of the same LLM are involved: the policy model and the reference model. The policy model is the LLM being trained: its weights update every DPO step. The reference model is a frozen version of that same LLM from before DPO began.

Because a language model outputs a probability distribution over tokens, it can score any response by computing the probability it would have generated that exact text. At each training step, both models score the chosen and rejected responses. A higher score means the model was more likely to have produced that text.

The reference model records how likely each response was before DPO. The loss compares the policy scores against the reference scores, and the policy's weights update so that rises $P(chosen)$ and $P(rejected)$ falls relative to the initial reference probabilities. This way, the model learns the preference without drifting far from where it started.

Note: the "score" is the probability the model would have generated that response.

One training step (recap):

Policy model scores the chosen response and the rejected response (given the prompt).
Reference model scores the same two responses.
The loss compares the two pairs of scores and updates the policy's weights:
$P(chosen)$ up, $P(rejected)$ down, relative to the reference.

DPO is a good choice when you have human preference data because it is simpler and cheaper than full RLHF. If you have preferred vs rejected answers: use DPO. If you have a reward model or need a flexible feedback loop: use RLHF.

Use cases

Open-source fine-tunes for smaller models: e.g., Zephyr fine-tuned Mistral-7B with DPO and approached ChatGPT-level chat quality on benchmarks.
Style and tone control: shifting how the model writes (e.g., tuning a support bot to be concise and on-brand).

Limitations

Offline: Model learns from the fixed dataset of good/bad answer pairs.
Overfits to dataset patterns: e.g., if chosen answers happen to be longer, it learns "longer = better."

Distillation

In distillation, a "teacher" model generates answers and a "student" model is trained via SFT to imitate them.

SFT and RL assume you already have good training data or a reward signal. Distillation is about generating that data. In short: SFT/RL define how to train and distillation defines where the training data comes from.

Off-policy distillation

A large "teacher" model generates answers to a set of prompts, and a smaller "student" model is trained via SFT on those answers. The teacher's outputs become the SFT dataset, and the student learns to imitate them. An example is when you want to shrink a big model into a smaller, cheaper one.

It's off-policy because the student trains on the teacher's trajectories, never its own.

Note: a trajectory is an ordered list of state-action pairs where a state is the text so far (prompt + everything generated), in tokens, and the action is the next token generated.

On-policy self-distillation (OPSD)

Off-policy distillation has one key limitation: there may be no stronger teacher available (e.g., when post-training a frontier model, there is no better model to learn from). OPSD solves this by having the student model create its own teacher signal.

OPSD can be done in two ways:

The teacher and the student are the same model, but the teacher gets privileged information. The student policy sees only the problem, while the teacher policy sees the problem plus extra context (e.g., a verified solution, an expert demonstration, an answer without the method, a rough method without the answer…)
Have the model generate many answers to the same prompt, use evals to select the best ones, then use SFT to train the model on those selected answers.

Note: the policy is the function used when predicting the next token.

OPSD's tradeoff: on-policy for the student, off-policy for the teacher

The teacher must score the student's sequences, which it would never have written itself. When the student commits to something the teacher knows is wrong, everything downstream degrades: the teacher, conditioned on a prefix contradicting its privileged info, stops modeling good answers and instead tries to explain the inconsistency.

Two mitigations we found:

Short outputs. Since one wrong token affects everything that follows, shorter outputs give less room for compounded error, which is why OPSD works better for short extraction-style JSON tasks than for long-form reasoning.

Forward KL over reverse KL. Say the student is working through a math problem and writes "12 + 15 = 34" and the real answer is 27.

Forward vs. reverse KL after a student mistake

After the mistake, at the next point where the model's reasoning has to use that result, the teacher's next-token distribution splits its probability across two groups of tokens: some tokens that would start steering the answer back on track (“recover,” shown as "wait" in the diagram), and some tokens that would keep going as if the wrong number were correct (go along with it, shown as "34"). Because continuing smoothly from what's already written is the easier, more natural move for a language model, the "go along with it" tokens usually get more total probability than the "recover" tokens. That's the left panel above.

Reverse KL trains the student to put almost all of its probability on whichever option the teacher favors most: it collapses onto a single answer. Since "go along with it" is favored, reverse KL pushes the student to put its probability there, fully committing to repeating the error. It's mode-seeking because it seeks out the single most likely token and dumps probability onto it, ignoring the rest. Middle panel.

Forward KL trains the student to put at least some probability on every token the teacher does, in roughly the proportions the teacher uses. So it still leans toward "go along with it" since that has more mass, but it doesn't zero out the "recover" tokens: some probability stays there too. That's mode-covering: it spreads probability to cover everywhere the teacher has any, rather than dumping it all on one token. Right panel.

Reverse KL locks the student into one choice, usually the one that repeats the mistake. Forward KL keeps recovery as an option, even if it's not the dominant one.

Which post-training technique should you use?

The right technique depends on the feedback you have. If you're teaching a new task or format, SFT is the natural starting point (with LoRA if cost-constrained). If you have human preference data, DPO would be a good choice since it's simpler and cheaper than full RLHF. When success can be verified automatically, like a correct answer or passing tests, RLVR with GRPO works best. And if the goal is a smaller, cheaper model, then you should use distillation.

In practice, these are stages in a pipeline rather than competing options: most production models go through SFT, then preference alignment, then RL on verifiable rewards. But if you know the failure cases and they converge to a few modes, harness-level changes or prompt engineering will probably be enough: SFT or RL isn't always necessary.

FAQ

Is DPO reinforcement learning? (DPO vs RL)

No. DPO is a supervised learning method: it optimizes a simple classification-style loss over static preference pairs, with no environment interaction, no sampling during training, and no reward model. It's derived from the RLHF objective mathematically, which is why it achieves similar alignment results, but the training procedure itself is closer to fine-tuning than to RL.

How much data do you need for supervised fine-tuning?

Often far less than people expect. For teaching format, tone, and instruction-following, roughly 1,000–10,000 high-quality examples can be enough, and quality beats quantity: a small, carefully curated expert dataset outperforms a large noisy one. With LoRA, small datasets are also less likely to degrade the model's general abilities, since 99% of the weights stay frozen.

What is catastrophic forgetting in fine-tuning?

Catastrophic forgetting is when fine-tuning too narrowly on one task causes a model to lose general capabilities it had before. Overfitting a small dataset makes responses repetitive and brittle, and low-quality examples teach the model low-quality habits. Common mitigations: LoRA (frozen base weights limit the damage), mixing general data into the fine-tuning set, and keeping learning rates low.

What is the difference between RLHF and RLAIF?

RLAIF (Reinforcement Learning from AI Feedback) replaces the human rankers in RLHF with an AI model that judges responses, typically guided by a written set of principles. The pipeline is otherwise the same: preferences train a reward model, and RL optimizes against it. RLAIF scales far more cheaply than human labeling and is the idea behind approaches like Anthropic's Constitutional AI.

What is reward hacking in RLHF?

Reward hacking happens when the policy learns to exploit weaknesses in the reward model rather than genuinely improving: producing responses the reward model scores highly but humans wouldn't actually prefer, like confident-sounding but wrong answers, excessive length, or flattery. It's a core reason RLHF uses a KL penalty against a frozen reference model, and a key motivation for verifiable rewards, where the signal can't be fooled as easily.