Layer normalization

Layer normalization is one of the three most common normalization strategies used in neural networks, the other two being batch normalization and group normalization. The name “layer normalization” is a bit of a misnomer, since you’re really normalizing across the features in an instance. To compute the layer normalization of an instance , we find the mean and variance across all features in . Then we subtract the mean and divide through by the standard deviation:

where is a small constant added for stability when the variance is small.