Learning from digits

From a half-adder to handwritten digits

In Lesson 5, we repeated backpropagation and used the learning rate to control each update. Now we use the same process to train a larger network to recognize handwritten digits.

What changes?

MNIST is a collection of labelled handwritten-digit images. Each image becomes 784 inputs, one for each pixel. The network has 32 hidden neurons and ten outputs, one for each digit from 0 to 9.

Then explore

See how an image becomes a prediction, train with batches and epochs, and check how well the model recognizes digits it did not train on.

Train a small MLP

784 inputs → 32 hidden neurons → 10 outputs
25,450 weights and biases · ReLU hidden layer

Learning rate Batch size

What is batch size?

A batch is a group of training images used for one model update. For example, a batch size of 32 means the model learns from 32 images together, not one at a time.

Within that batch of 32:

Use the current weights and biases to predict each image's digit and calculate its loss.
Backpropagate each image's loss to calculate its gradients. Keep the model unchanged while collecting them.
Average the gradients from all 32 images. Different images can push a parameter in different directions, so their contributions combine.
Multiply each average gradient by the learning rate, then subtract it from its weight or bias. This is one training step.

The next batch uses the updated model. One epoch finishes when all 50,000 training images have been used once.

With 32 images per batch, 50,000 images take 1,563 updates per epoch. The last batch has 16 images.

Smaller batches: more frequent updates, each based on fewer images. The gradients tend to vary more between updates.

Larger batches: fewer updates, each averaging more images. The gradients tend to vary less, but larger batches are not automatically faster or better at learning. Both sizes still use the same 50,000 images per epoch.

The Latest training batch score describes only that batch, not the whole training set.

Loading MNIST…

Steps: 0
Completed epochs: 0
Examples this epoch: 0 / 50,000

What happens in one step?

Run the batch through the network, calculate loss, and backpropagate. Average the gradients across the batch, multiply by the learning rate, then subtract them from the weights and biases.

An epoch finishes when every training example has been used once. We shuffle their order before the next epoch.

Starting values and loss

Hidden weights use He initialization: small random positive and negative values scaled to the number of inputs. Biases start at zero. The half-adder’s positive-weight start is not assumed to work best here.

The ten output scores become probabilities using softmax. Cross-entropy loss penalizes low probability for the correct digit. This replaces the squared-error loss from earlier lessons.

Inspect a prediction

Label: —

Hover a pixel to see its input value.

Prediction probabilities

Prediction: —

Latest training batchNot trained yet

Quick validation · 500 test digitsNot evaluated yet

Evaluation measures the model without changing its weights. Validation is checked after each epoch.

Training and evaluation data

Training: 50,000 digits. These examples produce the gradients that change the model.

Validation: 500 test digits. A fixed subset of the test set gives a quick, rough check after each epoch.

Test: 10,000 unseen digits. The full official MNIST test set never updates the weights.

This preview uses 50,000 of MNIST’s 60,000 training digits. Validation overlaps with test, not training. If you tune settings using the quick check or test mistakes, the test set is no longer an independent final check.