Loss Function - From Neuron to GPT

In the previous chapter, we built our first neural network model and used it to predict ice cream sales. But we don’t yet know whether that prediction is any good.

To judge a prediction, we first need to quantify its error. This is the job of the loss function: it measures the gap between the model’s predictions and the true values.

The loss function is at the heart of model training. It not only tells us “how wrong” the model is, but also provides the signal that guides the model to improve itself automatically (we will explore this over the next two chapters).

Error¶

The most direct way to quantify this is to subtract the prediction from the true value. The gap between them is called the error:

\text{error} = y - p

(1)

Where:

$y$ : the true value, called the label;
$p$ : the model’s prediction.

Squared error¶

Using the error directly has a problem: when we have multiple samples, positive and negative errors cancel each other out, so the overall error is underestimated. In practice, we usually use the square of the error, called the squared error:

\text{squared error} = (y - p)^2

(2)

The squared error has three advantages:

It removes the sign: every error becomes positive, so errors cannot cancel each other out;
It penalizes large errors: the larger the error, the heavier the penalty after squaring, which forces the model to fix big mistakes first;
It is differentiable everywhere: this is the mathematical prerequisite for the next chapter, where we use gradient descent to optimize the model parameters automatically.

💡 Tip: Why is being “differentiable everywhere” so important? Gradient descent needs the derivative of the loss function to decide in which direction to adjust the parameters. If the loss function is not differentiable at some point, the gradient cannot be computed there, and training fails.

Mean squared error (MSE)¶

When we have multiple samples (for example, sales data for $n$ days), we need an overall metric to measure the model’s performance. A common approach is to take the average of the squared errors of all samples, called the mean squared error (MSE):

\text{MSE} = \frac{1}{n} \sum\limits_{i=1}^{n} (y_i - p_i)^2

(3)

Where:

$y = [y_1, y_2, \ldots, y_n]$ : the $n$ true values;
$p = [p_1, p_2, \ldots, p_n]$ : the $n$ predictions.

The smaller the MSE, the more accurate the model’s predictions are overall. An MSE of 0 means the predictions match the true values exactly. That is almost impossible in practice, and not necessarily a good thing (it may indicate overfitting).

import numpy as np

Data¶

Now let’s return to Ming’s ice cream shop.

That evening, the sales figures come in: 165 ice creams were sold in total.

Features and labels¶

We call the true value the label and store it in a NumPy array as well. From now on, each data record consists of two parts:

Feature: the input data, i.e., the weather;
Label: the corresponding true value, i.e., the actual sales.

feature = np.array([28.1, 58.0])
label = np.array([165])

Model¶

We continue to use the model framework (parameters and inference function) from the previous chapter.

class Linear:

    def __init__(self, in_size, out_size):
        self.weight = np.ones((out_size, in_size)) / in_size
        self.bias = np.zeros(out_size)

    def __call__(self, x):
        return self.forward(x)

    def forward(self, x):
        return x @ self.weight.T + self.bias

Loss function (mean squared error)¶

Given the prediction $p$ and the label $y$ , the loss function computes the mean squared error. The result is called the loss.

The loss is a non-negative number:

the larger the loss, the larger the prediction error and the less accurate the model;
the smaller the loss, the closer the prediction is to the true value and the more accurate the model.

class MSELoss:

    def __call__(self, p, y):
        return self.loss(p, y)

    # Loss function (the end of forward propagation)
    def loss(self, p, y):
        # Loss
        return np.mean(np.square(y - p))

Modeling¶

When we create the model instance, we also create an instance of the loss function.

model = Linear(2, 1)
loss_fn = MSELoss()

Inference¶

prediction = model(feature)
print(f'prediction:\t{prediction}')

prediction:	[43.05]

Evaluation¶

With a loss function, we can quantify the error of the model’s inference.

loss = loss_fn(prediction, label)
print(f'loss:\t{loss}')

loss:	14871.802500000002

The loss is about 14,872. What does this number mean?

Because we use the squared error, the loss is measured in the square of the sales unit (ice creams²). It is not as intuitive as the error itself, but it is straightforward to compute:

the smaller the loss, the more accurate the model;
the trend of the loss is what matters: our goal is to make it keep decreasing during training.

The model predicted 43, and the true sales were 165, so the error is as large as 122. Since we only have one data record, the mean squared error is simply the squared error of that record: $122^2 = 14884$ , which roughly matches the loss of 14,872.

Our first model severely underestimates. So how do we make the model more accurate? That is exactly the question the next chapter starts to solve.

Exercises¶

Try changing the values of the weights and bias. Can you make the loss lower?