Measured at initialisation

Pre-LN vs post-LN: how big is the gradient?

Backprop through a LayerNorm multiplies the gradient by 1/σ, where σ is the rms of whatever that LayerNorm divides. Both stacks end in a LayerNorm. The difference is how big σ gets. Drag the depth and watch σ and the gradient move together.

The comparison comes from Xiong et al., On Layer Normalization in the Transformer Architecture (2020). They argue that at initialisation the gradient on the top layers is large for post-LN and about √L smaller for pre-LN, and use that to explain why post-LN needs learning-rate warmup and pre-LN can mostly skip it. The numbers below are measured, not taken from the paper.

post-LN x ← LN(x + F(x))

pre-LN x ← x + F(LN(x)), then one final LN

Depth L = 16

post-LNpre-LN

σ: rms of what the last LayerNorm divides

post

pre

Gradient on the top block's FFN weights, ‖∂L/∂W₂‖

post

pre

Pre-LN's top gradient is smaller by

Forward: what each LayerNorm has to divide

rms per element of the un-normalised stream leaving each block (1.0 = normalised)

Post-LN normalises after every block, so its LayerNorm only ever sees one block's worth of growth. Pre-LN never normalises the stream itself. Each block adds a roughly independent branch output, the variances add, and the rms climbs like √L.

Backward: gradient on each block's FFN weights

‖∂L/∂W₂‖_F per block

Post-LN stays between in every block at every depth. Pre-LN starts in that band at block 1 and falls as you go up. The dashed line is ÷ (pre-LN stream rms at that block, from the chart above): each block's gradient is set by the size of the stream it writes into.

Top block's gradient as the stack gets deeper

‖∂L/∂W₂‖_F of the last block, log-log. Click a depth to select it.

Post-LN stays near at every depth. Pre-LN falls with a log-log slope of for L ≥ 8. The dashed line is a 1/√L guide, the scaling in Xiong et al.'s bound, anchored at L = 64.

Where the measurement and the paper part ways

The √L shrink is not uniform across pre-LN blocks. Only the top block sits behind a stream of size √L. Block l is shrunk by its own stream size, about √l, and block 1 is not shrunk at all. Going down the stack, the residual branches add gradient back at the same rate the stream grew going up.
Xiong et al.'s Theorem 1 is an upper bound on the last FFN layer. It states O(d√ln d) and O(d√(ln d / L)), not Θ. Its claim about the other layers is a preliminary analysis.
√L is the asymptote. The stream starts at rms 1 and each block adds less than one unit of variance, so at L = 64 the measured shrink is , not √64 = 8×.

Check yourself: what does GPT-2's residual init do to the first chart?

GPT-2 (Radford et al., 2019) scales the init of the residual layers by 1/√N, where N is the number of residual layers. With an attention and an FFN branch per block, that's 1/√(2L) on every residual-branch output projection. Decide what that does to the pre-LN curve before opening the answer.

Answer

There are 2L branch writes, each with variance scaled by 1/(2L), so their total stays O(1) however deep the stack is. The stream stays O(1), the final LayerNorm's 1/σ stays O(1), and the top-block shrink goes away. This one is reasoning, not something measured on this page.

Table of the numbers

Setup: decoder-only Transformer, d = 256, 4 heads, FFN width 4d with ReLU, causal attention, Xavier-uniform weights, sequence length 32, batch 32, vocabulary 1000, random tokens and targets, mean cross-entropy. Each point is the mean over 10 seeds of one backward pass at init. After Xiong et al., On Layer Normalization in the Transformer Architecture (2020).