Is it actually learning?

Can it recognize digits it has not practiced on?

In Lesson 6, we trained a network to recognize handwritten digits. Now we compare how it does on its practice images and on separate images it never uses for training.

Train

Use practice images to adjust the weights and biases.

Validate

Check separate images as training progresses. Use these results to choose which saved model to keep.

Test

After choosing a model, check it on a third set of images. Keep these results out of your tuning decisions.

Choose the model’s layers, press Start, and compare the two curves. Then stop training and try the best saved model on the test set.

Workspace

Architecture & training

784 → 32 → 10 · 25,450 parameters · relu

Architecture

What do layers change?

Architecture means the model’s shape: how many hidden layers it has and how many neurons are in each. Hidden layers sit between the image inputs and the digit outputs. More neurons or layers give it more ways to learn patterns, but also more work to do.

Activation

What is sigmoid?

Sigmoid smoothly turns a node’s weighted sum into a value between 0 and 1. Larger sums give values closer to 1; smaller sums give values closer to 0. ReLU instead keeps positive sums and turns negative ones into 0.

Training images · keeps weights

Learning rate

Batch size

Loading MNIST…

Completed epochs: 0
Training steps: 0
Images processed: 0

What gets measured?

Accuracy is the percentage of digits guessed correctly. Loss measures how wrong the model’s probabilities are; lower is better.

Training accuracy this epoch counts every selected image as the model learns. These predictions come from different moments during training, not one final model.

The graph checks the same sample of training images and 1,000 validation images after each epoch. Validation images never change the weights. The final test checks 10,000 other images.

How does training work?

A batch is a group of images used for one weight update. An epoch is one pass through all the selected training images. The learning rate controls the size of each update.

Reset uses the same starting seed and image order so you can compare settings fairly.

Weights start as small random numbers; biases start at 0. With no hidden layers, the activation setting has no effect.

Stochastic gradient descent (SGD) learns from one mini-batch at a time. Average its gradients, multiply by the learning rate, then subtract from the weights and biases.

Cross-entropy is our loss. It checks the probability assigned to the correct digit: 90% gives a small loss, while 1% gives a large loss. A confident wrong answer is penalized strongly. We average this loss across the examples.

Training and validation

● Training check ● Validation ● Best validation loss

Waiting for the first measurement.

Training accuracy this epoch · 500 imagesNo images counted yet.

Training probe—

Validation probe—

Hover or tap a point to inspect it. Arrow keys work too.

How do I read the pattern?

For loss, lower is better. For accuracy, higher is better. Watch the trend across several epochs, not just one point.

Both improve: the model is learning patterns that also help on separate images. Only training improves: it may be learning details of its practice images that do not help on new ones. This is called overfitting.

Log epochs gives early epochs more space on the graph. It changes the spacing, not the results.

Layer sensitivity

Output → first hidden

The backward signal tells us how much changing a node affects the loss. These bars summarize its size in each layer. The longest bar is the 100% reference.

Why can more layers help?

Each layer builds on the previous layer’s results. For example, a network could combine small strokes into curves or loops, then use those patterns to distinguish digits. More layers give it more chances to combine simple patterns into complex ones.

But the learning signal must travel back through all those layers. If it becomes too small, the early layers learn slowly. Use these bars to watch the signal, and the validation curve to check whether extra layers actually help.

What does a fading signal mean?

At each sigmoid, the backward signal is multiplied by at most 0.25. Weights can increase it, but by the time it reaches the early layers closest to the input, the signal is often quite small. Those layers then receive very small weight changes and learn slowly. This is called vanishing gradients.

ReLU passes the signal through nodes whose saved forward sum was positive. It blocks it when that sum was zero or negative. The numbers beside the bars show signals too small to see.

Best checkpoint and test

A checkpoint is a saved copy of the model. Restore brings back the copy with the lowest validation loss.

Test checks that model on 10,000 separate images.

Lowest validation loss—

The test set does not choose the checkpoint.

Not tested yet.

Mistakes

Which digits get confused?

Each row is the correct digit; each column is the model’s guess. A number in row 3, column 8 counts how often it guessed 8 for a 3. Click a mistake with a count above zero to see an example.

Train first to inspect a validation mistake.