Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Fine-Tuning Techniques Survey

The base model obtained from pre-training can do only one thing: continue text. In Chapters 15 to 18, the supervised fine-tuning (SFT) we implemented taught it to converse in the “prompt → response” format, and direct preference optimization (DPO) made it prefer better responses. These two steps are a miniature of today’s post-training pipeline for LLMs.

But industrial post-training is far more complex: larger-scale instruction data, more memory-efficient fine-tuning methods, preference alignment based on reinforcement learning, and the most important breakthrough of the past two years: training reasoning models that “think” with verifiable rewards. In this chapter, we survey the mainstream techniques in fine-tuning and build a complete map of the field.

LLM technology evolves quickly. The content of this chapter is current as of November 2026.

Overview of fine-tuning techniques

A modern LLM, from base model to final product, usually goes through these stages:

StageDataLearning signalPurpose
Continued pre-training / mid-trainingLarge amounts of unlabeled textNext-token predictionAdd domain knowledge, extend the context size
Supervised fine-tuning (SFT)“Prompt + response” demonstrationsImitating demonstrationsLearn the conversation format and instruction following
Preference optimization (RLHF, DPO)Comparisons of better and worse responsesPreferencesAlign with human preferences: helpful, harmless, style
Reinforcement learning with verifiable rewards (RLVR)Problems with standard answersCorrect or wrong, judged by a programImprove reasoning, math, coding, and agent abilities
Distillation and mergingOutputs of a strong model, or multiple modelsImitation, parameter averagingTransfer abilities to small models, combine multiple models

Different models combine these stages differently. Take Qwen3 as an example: its post-training has four steps: cold start with long chain-of-thought data → reinforcement learning for reasoning → fusing the “thinking” and “non-thinking” modes → general reinforcement learning. Small models gain their abilities through “strong-to-weak” distillation. Llama 3, on the other hand, iterates over multiple rounds of “sampling → rejection sampling → SFT → DPO” and does not use PPO.

Advanced supervised fine-tuning

The superficial alignment hypothesis

Our SFT used only a few hundred samples. Surprisingly, this works to some extent. The LIMA study fine-tuned a 65B base model with only 1000 carefully selected samples and obtained strong conversational ability. This led to the superficial alignment hypothesis: almost all of a model’s knowledge and abilities come from pre-training, and fine-tuning mainly teaches the model which style and format to use in its responses.

The hypothesis has some truth, but it is incomplete:

So a more accurate statement is: the quality and diversity of the data matter more than its quantity, and different abilities need different amounts of data.

Instruction data

Where does instruction data come from?

The key points of data processing: deduplication, filtering out low-quality and harmful content, ensuring diversity of tasks and domains, and controlling the difficulty distribution. Also note that training a model on the outputs of other commercial models may be restricted by their terms of service.

Chat template

A multi-turn conversation must tell the model which segment is the system prompt, which is what the user said, and which is the assistant’s response. This is done by a chat template: special tokens mark the role and boundaries of each message. For example, the ChatML format wraps each message in <|im_start|> and <|im_end|>, while Llama 3 uses dedicated role header tokens.

A few key points:

Training tips

Continued pre-training

When a model needs to master a new domain (medicine, law, code) or a new language, the small amount of SFT data is not enough, and the model must be continually pre-trained on large amounts of unlabeled text. The training objective is exactly the same as in pre-training.

Key points:

Parameter-efficient fine-tuning

The cost of full-parameter fine-tuning

Full-parameter fine-tuning has to store many things for every parameter. With BF16 mixed precision and the Adam optimizer, each parameter needs: 2 bytes for the weight + 2 bytes for the gradient + 4 bytes for the FP32 master weight + 4 bytes each for Adam’s two moments, 16 bytes in total. A 7B model needs about 112 GB of GPU memory, not including activations, and a 70B model needs more than 1 TB.

In addition, every task needs a complete copy of the model, so serving multiple tasks makes storage and switching expensive.

LoRA

LoRA (Low-Rank Adaptation) is based on an observation: the weight change brought by fine-tuning roughly lies in a low-rank subspace. So we freeze the original weight WW and train only a low-rank increment:

W′=W+αrBA,A∈Rr×k, B∈Rd×r, r≪min⁡(d,k)W' = W + \frac{\alpha}{r} B A, \quad A \in \mathbb{R}^{r \times k},\ B \in \mathbb{R}^{d \times r},\ r \ll \min(d, k)
(1)

Where:

BB is initialized to 0, so at the start of training the model is identical to the original. The number of trainable parameters drops from dkdk to r(d+k)r(d+k), usually 1% of the original or less.

The benefits of LoRA:

Research from Thinking Machines (“LoRA Without Regret”) gave some practical conclusions:

QLoRA

QLoRA builds on LoRA by quantizing the frozen base weights to 4 bits (NF4 format) and training the adapters only in BF16, dequantizing the weights on the fly in the forward and backward passes. Combined with double quantization and paged optimizers, a single 48 GB GPU can fine-tune a 65B model. The cost is that dequantization adds extra computation, so training is slower than LoRA.

Reinforcement learning from human feedback

SFT can only make the model imitate demonstrations, but often “what is a good response” is hard to write as a demonstration and easy to compare: given two responses, people can easily tell which one is better. Reinforcement learning from human feedback (RLHF) exploits exactly this comparison information. The 1.3B model that InstructGPT trained with it beat the 175B GPT-3 in human evaluations.

Three steps

  1. Supervised fine-tuning: train on demonstration data to get the initial policy (which is also the reference model later);

  2. Training a reward model: collect human preference comparisons among multiple responses to the same prompt, and train a model to score responses;

  3. Reinforcement learning: let the model generate responses, score them with the reward model, and update the model with a reinforcement learning algorithm so that it gets higher scores.

Reward model

A reward model is usually a language model with an extra head that outputs a scalar. It is trained with the Bradley-Terry model: assume that the probability that response ycy_c is better than yry_r depends on the difference of their rewards:

P(yc≻yr∣x)=σ(r(x,yc)−r(x,yr))P(y_c \succ y_r \mid x) = \sigma\big(r(x, y_c) - r(x, y_r)\big)
(2)

Where:

The training loss is −log⁡σ(r(x,yc)−r(x,yr))-\log \sigma\big(r(x, y_c) - r(x, y_r)\big), which makes the preferred response score higher.

A reward model is only an imperfect proxy for human preferences. The model exploits its loopholes: it generates responses that are not truly better but score higher with the reward model, such as longer and more flattering ones. This is called reward hacking or over-optimization.

Reinforcement learning

The optimization objective of the reinforcement learning stage is:

max⁡π Ex, y∼π[r(x,y)]−β KL(π ∥ πref)\max_{\pi} \ \mathbb{E}_{x,\, y \sim \pi}\big[r(x, y)\big] - \beta\, \mathrm{KL}\big(\pi \,\|\, \pi_{\text{ref}}\big)
(3)

Where:

The first term maximizes the reward, and the second term uses the KL divergence to penalize the policy for drifting too far from the reference model. Its role is to keep the responses fluent and to limit the room for reward hacking. The larger β\beta is, the stronger the constraint.

The most commonly used algorithm is PPO (Proximal Policy Optimization):

In a language model, each token is an “action”, and the reward is usually given only at the end of the whole response.

The cost of RLHF

The relationship between DPO and RLHF

DPO starts from the discovery that the optimization objective above has a closed-form optimal solution:

π∗(y∣x)=1Z(x) πref(y∣x) exp⁡ ⁣(r(x,y)β)\pi^*(y \mid x) = \frac{1}{Z(x)}\, \pi_{\text{ref}}(y \mid x)\, \exp\!\Big(\frac{r(x, y)}{\beta}\Big)
(4)

Where:

Conversely, the reward can be expressed in terms of the policy:

r(x,y)=βlog⁡π∗(y∣x)πref(y∣x)+βlog⁡Z(x)r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)
(5)

Substituting it into the Bradley-Terry preference probability, the term Z(x)Z(x), which does not depend on yy, cancels exactly in the difference between the two responses. The reward model is " eliminated", and we get a loss that depends only on the policy and the reference model:

LDPO=−log⁡σ(β[log⁡πθ(yc∣x)πref(yc∣x)−log⁡πθ(yr∣x)πref(yr∣x)])\mathcal{L}_{\text{DPO}} = -\log \sigma\Big(\beta\Big[\log\frac{\pi_\theta(y_c \mid x)}{\pi_{\text{ref}}(y_c \mid x)} - \log\frac{\pi_\theta(y_r \mid x)}{\pi_{\text{ref}}(y_r \mid x)}\Big]\Big)
(6)

Where:

This is the DPO we implemented. So DPO is not a method completely different from RLHF, but an offline, simplified way of solving the same optimization objective: no reward model, no online sampling, and no value model, and it trains just like SFT. The quantity in the brackets can be seen as the model’s implicit reward, and β\beta is the strength of the KL constraint.

Other preference optimization algorithms

Problems with DPO

DPO is simple and effective, but it has some known problems:

Online preference optimization

The fundamental limitation of offline methods is that the data does not come from the current policy. Online or iterative methods let the model generate new data while it trains:

AI feedback

Human annotation is expensive and slow, so AI is increasingly used to provide feedback (RLAIF):

AI judges have biases of their own: they prefer longer responses, responses placed first, and responses in their own style. So human data is still needed for calibration.

Reinforcement learning with verifiable rewards

Verifiable rewards

The reward of RLHF comes from a trained reward model that is easy to exploit. For tasks with objective answers, the reward can be computed directly by a program:

This is reinforcement learning with verifiable rewards (RLVR), a name proposed by Tülu 3. Its benefits are that the reward is objective, no reward model has to be trained, and it is hard to hack. Its limitation is that it applies only to tasks whose correctness can be verified automatically.

GRPO

GRPO (Group Relative Policy Optimization) was proposed by DeepSeekMath, and it removes the value model from PPO. The approach: for each prompt, sample a group of (GG) responses and use the relative reward within the group as the advantage:

A^i=ri−mean(r1,…,rG)std(r1,…,rG)\hat{A}_i = \frac{r_i - \text{mean}(r_1, \dots, r_G)}{\text{std}(r_1, \dots, r_G)}
(7)

Where:

Within a group, responses better than average are encouraged, and those worse than average are suppressed. The optimization objective is still a PPO-style clipped objective: for each token, compute the probability ratio between the new and old policies and multiply it by this advantage, usually with a KL penalty against the reference model added.

Removing the value model saves a lot of memory and engineering complexity, and there is no longer a value function that is hard to train. GRPO has therefore become one of the mainstream algorithms for training reasoning models.

DeepSeek-R1

DeepSeek-R1, released in January 2025, made RLVR the focus of attention:

This pipeline has become the standard practice for reasoning models. Later models such as Qwen3 also adopted a similar “cold start → reasoning RL → general RL” pipeline.

Variants and improvements

GRPO exposed some problems in large-scale training with long chains of thought, which gave rise to a batch of improved methods:

Another class of problems comes from the system: the training engine and the inference engine are not numerically identical, which quietly turns supposedly “online” sampling into “offline” sampling. A common fix is truncated importance sampling.

Beyond math and code, RLVR has been extended to agent scenarios: the model completes tasks in multi-turn environments where it can use tools (for example, fixing a bug in a code repository), and whether the task is completed serves as the reward. DeepSeek-V4 (released in April 2026) is one example: it targets long-running agent tasks and is trained with reinforcement learning in real tool environments.

What reinforcement learning actually brings is still debated: some studies found that RLVR mainly raises the probability that the model gets a problem right in one attempt, but with a large number of samples, the range of problems the base model can solve has not clearly expanded. This remains an active research topic.

Process rewards

The rewards above look only at the final answer and are called outcome rewards. A process reward model (PRM) scores every step of the reasoning:

There are also quite a few methods for building process labels automatically (for example, estimating the value of each step with Monte Carlo sampling, or learning it implicitly from outcome rewards). But the DeepSeek-R1 report mentions that their attempts with process rewards had limited effect. Mainstream reasoning models still rely mainly on outcome rewards.

Knowledge distillation and model merging

Knowledge distillation

Knowledge distillation lets a small model (the student) learn the behavior of a large model (the teacher):

When using distillation, pay attention to the license of the teacher model.

Model merging

Model merging needs no training and directly combines multiple models into one in parameter space:

Merging requires the models to come from the same base model, and the result is not always guaranteed, so it needs evaluation. The open-source community often uses tools such as mergekit.

Training systems

Memory optimization

The memory of fine-tuning comes mainly from weights, gradients, optimizer states, and activations. Common ways to save memory:

Training systems for reinforcement learning

Reinforcement learning training is completely different from SFT: every round first lets the model generate a large number of responses and then updates the model with them. Generation often takes most of the time, and the lengths of the responses vary widely, so a few very long responses slow down the whole batch.

So a reinforcement learning system usually has two parts:

The latest weights must be synchronized between the two. There are two deployment modes: colocated (the same GPUs run generation and training in turns, saving GPUs) and disaggregated (generation and training use different GPUs and can run asynchronously, but generation uses a slightly older model, so data staleness has to be handled).

In addition, computing the reward is itself a systems problem: for example, running code concurrently in sandboxes, calling verifiers, and managing the environments of multi-turn agents. The numerical differences between the training and inference engines must be handled, as mentioned in the inference survey, with batch-size-invariant kernels and importance correction.

Common tools

PyTorch’s official torchtune is no longer actively maintained (its documentation says development wound down in 2025), so it is not recommended for new projects. These tools update quickly, so when choosing, look at how active they are and how well they support new algorithms.

Evaluation

A fine-tuned model is evaluated along many more dimensions than a base model:

Evaluation has its own pitfalls: biases of judge models, contamination and saturation of benchmarks, and reward hacking in training, where the optimized metric improves but the real quality does not. So evaluation should use multiple complementary metrics and include human inspection wherever possible.

In this chapter, we surveyed the mainstream techniques in fine-tuning:

The SFT and DPO we implemented from scratch are the starting point of this chain: SFT teaches the model the format through demonstrations, and DPO teaches the model preferences through comparisons. Reinforcement learning moves the model from “imitating human responses” to “finding solutions by trial and error”, which is one of the fastest-improving directions of LLM capability today.

References

Fine-tuning
Direct Preference Optimization: Model Training