What is "early stopping" in neural network training?

MLP (multilayer perceptron) neural networks are very flexible mathematical functions of their inputs, making it very easy to overfit the data. To avoid overfit, it is therefore necessary to somehow constrain the modeling process. One way to do this is by early stopping of training.

Typically, a MLP begins training with a poor fit to the data (due to its random weight initialization). As training progresses the neural networks fit to the data improves. At some point, however, the neural network begins to overfit the data, meaning that its performance on the training data continues to improve, but only because it is beginning to memorize the peculiarities of the training cases, not because it is learning more about the underlying process. Remember, the object is to have the neural network generalize usefully to new cases, not memorize the training cases.

Early stopping involves training the MLP with some fraction of the data, and testing periodically with a seperate set of data (let's call it the "test data"). Performance on the training data (called the "apparent error") will usally only trend downward. Performance as tested on the test data (an estimate of the "true error") will typically go down like the the apparent error, but eventually diverge and, at some point, start to become worse. The best place to stop training is indiciated by the minimum estmated true error (that is, the best performance on the test data). Good commercial neural network software will handle all of this automatically, saving the network weights with the best estimated true error.

There are other ways to guard against overfitting, such as limiting the number of hidden nodes, but early stopping is fairly easy to implement and use.