Label smoothing

Label smoothing is a technique used in deep learning to regularize the training process of a neural network. The technique is used to avoid overfitting by reducing the reliance of the model on correct labels during the training process. It works by increasing the confidence of the model in “wrong” labels.

During training, a neural network will produce a probability distribution over the correct labels for a given input. Label smoothing adds a small amount of confusion to the output of the network. This confusion encourages the model to consider “incorrect” labels as well, reducing the likelihood of overfitting.

Label smoothing is especially useful when training a network on data that has large discrepancies between classes. For example, a network trained on a dataset with 100 positive samples and 1 negative sample would be likely to become very overfitted to the positive sample after a few training passes. Applying label smoothing would reduce the certainty with which the model chooses the positive label, increasing its generalization.

Label smoothing can be applied in two ways. The first is to apply a constant amount of smoothing on every label. This is achieved by adding a small portion of the uniform distribution on every label. The other way is to apply varying levels of smoothing on each label, depending on how certain the network is on the correct classification. For example, a label that is very certain would have a high level of smoothing applied, whereas a label that is more uncertain would have less smoothing.

Label smoothing is a commonly used technique for improving the performance of deep learning models. Its use in combination with other regularization techniques, like early stopping and dropout, can help to further reduce overfitting and improve model generalization.