Inside AI · Topic 5 of 10

Training and testing a model

Data is split into training, validation and test sets. We measure accuracy and loss, and watch out for overfitting and underfitting.

📌 Key points

Data is usually split into three sets: a training set (the model learns from it), a validation set (to check progress and tune settings) and a test set (a final check on data never seen).
Most data goes to training, e.g. about 70% training, 15% validation and 15% test.
Accuracy = correct predictions ÷ total predictions, often given as a percentage.
Loss is a number that measures how wrong the predictions are. Training tries to make the loss as small as possible.
One full pass through all the training data is called an epoch. Training often takes many epochs.
Overfitting: the model learns the training data too closely, even its mistakes and odd details. It scores well in training but badly on new data, like memorising practice answers.
Underfitting: the model is too simple or has not trained enough. It does badly even on the training data.
Good data matters: more examples, many different kinds, correct labels and a fair mix. "Garbage in, garbage out."

Training set: The data the model learns from.
Validation set: Data used during development to check progress and choose settings.
Test set: New data kept back until the end, to check how well the model really works.
Accuracy: The fraction (or percentage) of predictions that are correct.
Loss: A number that measures how wrong the model's predictions are.
Overfitting: When a model learns the training data too closely and does badly on new data.
Underfitting: When a model is too simple to learn the patterns, even in its training data.

A model scores 99% accuracy on its training data but only 60% on the test set. What is going on, and what can be done?

There is a big gap between training (99%) and test (60%), so the model does well on examples it has seen but not on new ones.
This is overfitting: it has memorised the training examples instead of learning general patterns.
Fixes: collect more, and more varied, training data; use a simpler model; or stop training earlier, when the validation score stops improving.
After the changes, check again on the test set. A good model scores well on both.

💡 Remember

Remember: never judge a model only on the data it trained on. The real test is new data.

Test yourself8 questions from a bank of 19. A new mix every time.