Gradient Descent - From Neuron to GPT

In the previous two chapters, we built the basic framework of the model:

Forward propagation: compute the prediction from the model parameters (the weights and the bias);
Loss function: quantify the error between the prediction and the true label.

The model parameters so far were initialized by hand, and as the inference and evaluation results showed, they are not accurate. The core question we now face is: how do we adjust the model parameters to make the loss smaller?

The answer is to let the network model learn from data on its own. This process is called model training.

Model training¶

The goal of model training is clear: find a set of model parameters (weights and bias) that makes the loss as small as possible.

This is a classic mathematical problem: function optimization. An equally classic analogy: imagine the loss as a mountain. The goal of training is to find the valley, the place where the loss is smallest.

Neural networks use an optimization method called gradient descent. The idea is intuitive: standing on a mountain, take a step in the steepest downhill direction each time, and you will eventually reach the valley.

More concretely, if we substitute the inference function into the loss function, we find that the loss $L$ is a function of the weight $w$ and the bias $b$ , written $L(w, b)$ .

The gradient is the direction of “steepest ascent” of this function. In mathematical terms, it is the vector of partial derivatives:

\nabla L = \left( \frac{\partial L}{\partial w},\ \frac{\partial L}{\partial b} \right)

(1)

Descent means moving the parameters one step in the opposite direction of the gradient. Because the gradient points uphill, the opposite direction naturally points downhill:

w_{\text{new}} = w_{\text{old}} - \frac{\partial L}{\partial w}

(2)

b_{\text{new}} = b_{\text{old}} - \frac{\partial L}{\partial b}

(3)

Computing the gradient¶

The gradient is the partial derivative of the loss function with respect to the model parameters. There are three ways to compute it:

Numerical differentiation¶

Add a tiny perturbation $\epsilon$ to each parameter and observe how the loss changes to (approximately) compute the derivative:

\frac{\partial L}{\partial w_i} \approx \frac{L(w_i + \epsilon) - L(w_i)}{\epsilon}

(4)

In model training this is extremely expensive: with as many parameters as the model has, we need that many forward propagations. A model with a million parameters would need a million inferences, which is completely unacceptable.

Symbolic differentiation¶

Derive the derivative formula of each parameter by hand. This is fine for simple linear regression, but a real network can have dozens of layers and all kinds of nonlinear transformations, so a hand-derived formula can run to several pages and the expression explodes. Worse, every time the network structure changes, all the formulas have to be derived again from scratch.

Automatic differentiation¶

Use the chain rule of calculus. The core idea is that any complex computation can be decomposed into a combination of simple basic operations, and the derivative rule of each basic operation is fixed and known. Decompose the forward propagation into basic operations, and we can apply the chain rule backward to compute the gradients of all the parameters step by step.

The loss $L$ is a function of the prediction $p$ (the loss function), and the prediction $p$ is in turn a function of the model parameters $w$ and $b$ (the inference function). By the chain rule of calculus, we can “chain” their derivatives together to get the derivative of the loss with respect to the parameters:

\frac{\partial L}{\partial w} = \frac{\partial L}{\partial p} \cdot \frac{\partial p}{\partial w}, \qquad \frac{\partial L}{\partial b} = \frac{\partial L}{\partial p} \cdot \frac{\partial p}{\partial b}

(5)

Now let’s use the chain rule to derive the gradient formulas.

Step 1: compute the derivative of the loss function with respect to the prediction. The loss function (for a single sample) is $L = (y - p)^2$ . Differentiating with respect to $p$ :

\frac{\partial L}{\partial p} = -2(y - p)

(6)

Step 2: compute the derivative of the inference function with respect to the parameters. The inference function is $p = w \cdot x + b$ . Differentiating with respect to $w$ and $b$ respectively:

\frac{\partial p}{\partial w} = x, \qquad \frac{\partial p}{\partial b} = 1

(7)

Step 3: combine them with the chain rule to get the complete gradients of the loss $L$ with respect to $w$ and $b$ :

\frac{\partial L}{\partial w} = \frac{\partial L}{\partial p} \cdot \frac{\partial p}{\partial w} = -2(y - p) \cdot x

(8)

\frac{\partial L}{\partial b} = \frac{\partial L}{\partial p} \cdot \frac{\partial p}{\partial b} = -2(y - p)

(9)

The common part of these two gradient formulas, the derivative of the loss function with respect to the prediction, $-2(y - p)$ , has a special name in deep learning: the error term (also called delta), written $\delta$ :

\delta = -2(y - p)

(10)

With the error term, the gradients can be written concisely:

\frac{\partial L}{\partial w} = \delta \cdot x, \qquad \frac{\partial L}{\partial b} = \delta

(11)

The meaning of the error term $\delta$ is intuitive. If $\delta < 0$ , the model’s prediction is too low; if $\delta > 0$ , the model’s prediction is too high. Gradient descent uses this signal to correct the parameters.

import numpy as np

Data¶

feature = np.array([28.1, 58.0])
label = np.array([165])

Model¶

Gradient descent has two steps: gradient computation and parameter update. We first add the part that adjusts the model parameters according to the error term to the network model, while the gradient computation is done in the loss function.

Backward function¶

The backward function uses the error term $\delta$ (computed by the loss function) to update the model parameters:

w \leftarrow w - \delta \cdot x

(12)

b \leftarrow b - \delta

(13)

This flow of data from the output toward the input is called backpropagation.

class Linear:

    def __init__(self, in_size, out_size):
        self.weight = np.ones((out_size, in_size)) / in_size
        self.bias = np.zeros(out_size)

    def __call__(self, x):
        return self.forward(x)

    def forward(self, x):
        return x @ self.weight.T + self.bias

    # Backward function (backpropagation, update the parameters)
    def backward(self, d, x):
        self.weight -= d * x
        self.bias -= np.sum(d)

Loss function (mean squared error)¶

The error term $\delta$ is computed inside the loss function.

Gradient function¶

Based on the derivation above, the gradient function computes the error term $\delta$ from the label $y$ and the prediction $p$ :

\delta = -2(y - p)

(14)

The error term will be used in the model’s backward function to compute the weight gradient $\delta \cdot x$ and the bias gradient $\delta$ .

class MSELoss:

    def __call__(self, p, y):
        return self.loss(p, y)

    def loss(self, p, y):
        return np.mean(np.square(y - p))

    # Gradient function (the starting point of backpropagation)
    def gradient(self, p, y):
        # Error term
        return -2 * (y - p)

Modeling¶

model = Linear(2, 1)
loss_fn = MSELoss()

Training¶

Now let’s train the model for the first time with gradient descent. The training process consists of:

Forward propagation: infer the prediction from the features and the model parameters;
Error computation: compute the error term from the prediction and the label;
Backpropagation: use the error term to compute the gradients, and then adjust the model parameters.

prediction = model(feature)
delta = loss_fn.gradient(prediction, label)
model.backward(delta, feature)

Inference¶

After training, we run inference again with the adjusted model parameters.

prediction = model(feature)
print(f'prediction:\t{prediction}')

prediction:	[1013352.429]

Evaluation¶

loss = loss_fn(prediction, label)
print(f'loss:\t{loss}')

loss:	1026548766283.6302

The loss exploded from 14,872 to 1,026,548,766,283, an increase of nearly 70 million times.

What happened?

The prediction jumped from 43 to 1,013,352 in a single step, completely overshooting the true value 165 and landing far on the other side of the valley. It is like taking a step so large while walking downhill that you land on the opposite slope, even higher than where you started.

In deep learning, this phenomenon is called divergence: the loss not only fails to decrease but grows sharply, and training fails.

💡 Tip: In later chapters, we will introduce the learning rate to control the size of each parameter adjustment, which solves the problem of diverging gradients during training.

Exercises¶

If the prediction is larger than the true value (the prediction is too high), will the error term $\delta$ be positive or negative? In which direction will the parameters be updated? Think about it using the formulas.