In the first ten chapters, we completed the pre-training of the GPT model: the model learned the statistical patterns of language and can continue text in Shakespeare’s style.
But pre-training leaves a gap: it only taught the model to talk to itself, not to converse with people. If you say “Hello!”, it may simply continue your words with “The weather is nice today!” instead of answering “Hello! How can I help you?”.
In practice, researchers found that providing a small number of conversation templates for the model to keep learning from is enough to switch it easily from continuation mode to conversation mode. This step is called fine-tuning, and it has become a standard part of GPT training.
Supervised fine-tuning¶
Supervised fine-tuning (SFT) is the most basic fine-tuning method and the first step in fine-tuning large GPT models:
Pre-training: self-supervised learning that learns the patterns of language from massive amounts of text. The data needs no manually annotated labels;
Supervised fine-tuning: supervised learning that learns the conversation pattern from (prompt, response) pairs. The data must be constructed by people.
Data format¶
The data for supervised fine-tuning consists of (prompt, response) pairs:
Prompt: the user’s question or command;
Response: the ideal answer from the model.
For example:
Prompt: ROMEO:
Hello, Juliet!
Response: JULIET:
Hello, Romeo!Data source¶
The corpus for the supervised fine-tuning stage should be new, annotated data that is specific to the target task or domain and was not used during pre-training.
We will extract conversation samples from the test set of the tiny Shakespeare (tinyshakespeare) dataset, which guarantees that the model has not seen this data during pre-training.
💡 Tip: To make teaching easier, the fine-tuning samples come from the same Shakespeare script as pre-training, using only the test-set part that pre-training did not use, so that no new dataset has to be prepared. In real projects, fine-tuning data should be collected specifically for the target task, and its source and style usually differ from those of the pre-training corpus.
The format of the script is:
SPEAKER A:
CONTENT...
SPEAKER B:
CONTENT...We pair two adjacent speeches, using the former as the prompt and the latter as the response, and this builds the SFT samples.
import pickle
import re
from itertools import pairwise
import numpy as np
from src.gpt import *np.random.seed(42)Data extraction¶
The extraction function extract_turns extracts all the speeches from the test set of the Shakespeare script (the last 10% of the speech blocks, which lies inside the test set used in pre-training):
Splitting: split the script into speech blocks on blank lines;
Identifying the speaker: the first line of each block matches the
SPEAKER:format, from which the speaker is extracted;Extracting the speech: merge the remaining lines, keeping whole lines, into the speech content, with a total length of at most
max_turn_chars(100 characters).
The function returns a list in which each element is a (speaker, speech) pair.
def extract_turns(filename, split=0.9, max_turn_chars=100):
with open(filename, encoding="utf-8") as f:
text = f.read()
# Split on blank lines; each block is one speech by one character
blocks = re.split(r"\n\s*\n", text.strip())
turns = []
# Take the test set
split = int(len(blocks) * split)
for block in blocks[split:]:
lines = block.strip("\n").split("\n")
if not lines:
continue
# Format check: the first line must be "SPEAKER:", otherwise this is not a speech block
m = re.compile(r"^([A-Z][A-Za-z' ]{0,30}):\s*$").match(lines[0].strip())
if not m:
continue
speaker = m.group(1).strip()
# Keep the original line breaks and take whole lines
content_lines = []
length = 0
for line in lines[1:]:
line = line.strip()
if not line:
continue
if length + len(line) + 1 > max_turn_chars:
break
content_lines.append(line)
length += len(line) + 1
if not content_lines:
continue
turns.append((speaker, "\n".join(content_lines)))
return turnsHyperparameters¶
We use only one hyperparameter: the context size, which must stay the same as in pre-training.
CONTEXT_SIZE = 256DATA_FILE = "../../../../data/tinyshakespeare.txt"
dataset = CharDataset(DATA_FILE, 1, CONTEXT_SIZE)Extracting the samples¶
Build (prompt, response) pairs from the test set of the dataset:
Pairing: use
itertools.pairwise()to pair adjacent speeches, with the former as the prompt and the latter as the response;Shuffling: shuffle the order of the samples randomly;
Truncation: take the first 512 samples;
Encoding: encode the prompt and the response into token sequences separately.
# Pair adjacent speeches
pairs = list(pairwise(extract_turns(DATA_FILE)))
# Randomly pick 512 pairs
np.random.shuffle(pairs)
pairs = pairs[:512]
samples = []
for (speaker_a, content_a), (speaker_b, content_b) in pairs:
# Prompt
prompt = f"{speaker_a}:\n{content_a}\n\n"
# Response
response = f"{speaker_b}:\n{content_b}\n\n"
samples.append((dataset.encode(prompt), dataset.encode(response)))Saving the samples¶
Save them to a file with pickle.dump().
SFT_SAMPLES = "../../../../data/sft-samples.pkl"
with open(SFT_SAMPLES, "wb") as f:
pickle.dump(samples, f)
print(f"saved {len(samples)} SFT samples to {SFT_SAMPLES}")saved 512 SFT samples to ../../../../data/sft-samples.pkl
Verifying the samples¶
Look at the first sample to confirm that the format is correct.
sample = samples[0]
print(f"{dataset.decode(sample[0])}\n{dataset.decode(sample[1])}")Pedant:
Away, away, mad ass! his name is Lucentio and he is
VINCENTIO:
Lucentio! O, he hath murdered his master! Lay hold
on him, I charge you, in the duke's name. O, my
In this chapter, we extracted 512 conversation samples from the test set of the tiny Shakespeare dataset and saved them as SFT training data. Each sample is a (prompt, response) pair.
In the next chapter, we will use these samples to perform supervised fine-tuning on the pre-trained model, so that it learns to answer a prompt in the conversation format.
Exercises¶
Compare the data pairs (features: labels) in the pre-training and supervised fine-tuning stages. What is different, and why?