Gradient clipping

How norm clipping works

A gradient describes how changing a model parameter would change the training loss. A very large gradient can produce an excessively large update. Norm clipping places a ceiling on the combined gradient magnitude.

If the gradient norm is above the chosen ceiling, scale the gradients down together. Otherwise, leave them unchanged. Scaling all components by the same factor preserves the direction of the gradient vector.

For example, a gradient vector with norm 12 and a ceiling of 3 is scaled to one quarter of its original size. Its new norm is 3. This is an illustrative calculation, not a recommended threshold for every model.

Norm clipping and value clipping

Norm clipping bounds the size of the whole gradient vector. Value clipping bounds each component separately, which can change the vector's direction. Specify which method an experiment uses; they produce different updates.

PyTorch's clip_grad_norm_ computes a total norm across the supplied parameter gradients and modifies those gradients in place. The configured max_norm sets the ceiling.

What to monitor

Apply clipping after computing gradients and before the optimizer update. Track the original gradient norm, how often clipping occurs, training loss, and evaluation results. If nearly every step is clipped heavily, investigate the learning rate, data, and training setup.

Choose a threshold through experiments on the actual training workload. Clipping can control large updates, but a stable loss curve alone does not establish that a coding model produces useful changes. Evaluate completion quality and tests as well.