Attention in the y = abx Toy Model

Learning dynamics of a scalar linear attention layer on in-context regression

post.cover
Kinkaku-ji, Kyoto, Japan

This is joint work with James Ang Ming Liang, and a follow-up to his post on training dynamics. There he studies gradient descent on the toy model $f(x) = abx$, which is two scalar weights multiplied together. Even that small model has a whole manifold of solutions, a conserved quantity, and a saddle at the origin.

Here we ask what the analogous toy model for attention looks like.

Setup

As in the $abx$ model, every token and every weight is a single number. The input is a sequence of $n$ context tokens $(x_1, y_1), \dots, (x_n, y_n)$, followed by a query $x_q$. The model has to produce a prediction $\hat{y}$ for the query.

Give the layer a scalar key weight $k$, query weight $q$ and value weight $v$. Each context token offers key $k x_i$ and value $v y_i$. The query is $q x_q$. Linear attention averages value times key times query:

\[\hat{y} = \frac{1}{n} \sum_{i=1}^{n} (v\, y_i)(k\, x_i)(q\, x_q).\]

There is no softmax here, so the weight on each value is the raw product of its key with the query.

In-context regression

Each sequence has its own hidden slope $\theta$. The context shows $n$ examples with $y_i = \theta x_i$, and the model must output $\theta x_q$ for the query. The slope changes from sequence to sequence, so the only way to do well is to read $\theta$ off the context. We draw $\theta, x_i, x_q \sim \mathcal{N}(0, 1)$ independently.

Plugging $y_i = \theta x_i$ into the model gives

\[\hat{y} = \frac{1}{n} \sum_{i=1}^{n} (v\, \theta x_i)(k\, x_i)(q\, x_q) = \underbrace{(v\, k\, q)}_{p}\, \theta\, S\, x_q, \qquad S = \frac{1}{n} \sum_{i=1}^{n} x_i^2.\]

Only the product $p = vkq$ matters, and $S$ is the average of $n$ squared normal numbers, which is close to 1 for long contexts.

Closed form of the loss

We train with squared error, averaged over sequences:

\[L(p) = \tfrac12\, \mathbb{E}\big[(\hat{y} - \theta x_q)^2\big].\]

This has a closed form:

\[L(p) = \frac{n+2}{2n}\,(p - p^*)^2 + \frac{1}{n+2}, \qquad p^* = \frac{n}{n+2}.\]

So the best product is $p^* = \frac{n}{n+2}$, and even there the loss cannot go below $\frac{1}{n+2}$.

The plot below shows this loss as a function of $p$. The line is the closed form, and the circles are the average loss of the model’s predictions on 30,000 sampled sequences. Drag the slider to change the context length.

4
closed form sampled best p and the loss floor
best product p* = n/(n+2) =    loss floor 1/(n+2) =

With a short context the best product is well below 1 and the loss cannot reach zero. As $n$ grows, $p^*$ moves toward 1 and the floor goes to zero.

Derivation

Step 1. Write the error for one sequence. The prediction is $\hat{y} = p\,\theta S x_q$ and the target is $\theta x_q$. Factor out what they share:

\[\hat{y} - \theta x_q = \theta x_q\,(pS - 1).\]

So the loss on one sequence is

\[\tfrac12(\hat{y} - \theta x_q)^2 = \tfrac12\,\theta^2\,x_q^2\,(pS-1)^2.\]

Step 2. Average, using independence. $\theta$, $x_q$ and the $x_i$ are all drawn independently, and $S$ depends only on the $x_i$. The expectation of a product of independent things is the product of expectations:

\[L(p) = \tfrac12\;\mathbb{E}[\theta^2]\;\mathbb{E}[x_q^2]\;\mathbb{E}\big[(pS-1)^2\big].\]

Both $\theta$ and $x_q$ are standard normal, so $\mathbb{E}[\theta^2] = \mathbb{E}[x_q^2] = 1$ and they drop out:

\[L(p) = \tfrac12\,\mathbb{E}\big[(pS-1)^2\big] = \tfrac12\Big(p^2\,\mathbb{E}[S^2] - 2p\,\mathbb{E}[S] + 1\Big).\]

So we only need two numbers, $\mathbb{E}[S]$ and $\mathbb{E}[S^2]$.

Step 3. Compute those two numbers. Recall $S = \frac1n\sum_i x_i^2$.

For the mean, each $\mathbb{E}[x_i^2] = 1$, so $\mathbb{E}[S] = 1$.

For the second moment, use $\mathbb{E}[S^2] = \mathrm{Var}(S) + (\mathbb{E}[S])^2 = \mathrm{Var}(S) + 1$. Since $S$ is an average of $n$ independent terms,

\[\mathrm{Var}(S) = \frac{1}{n^2}\cdot n\cdot \mathrm{Var}(x_i^2) = \frac{\mathrm{Var}(x_i^2)}{n}.\]

A standard normal has fourth moment $\mathbb{E}[x^4] = 3$, so

\[\mathrm{Var}(x_i^2) = \mathbb{E}[x_i^4] - \big(\mathbb{E}[x_i^2]\big)^2 = 3 - 1 = 2.\]

Therefore $\mathrm{Var}(S) = \frac2n$ and

\[\mathbb{E}[S^2] = 1 + \frac2n = \frac{n+2}{n}.\]

(The fourth moment comes from integration by parts on the Gaussian density, which gives $\mathbb{E}[x^{k+1}] = k\,\mathbb{E}[x^{k-1}]$. With $k = 3$ that is $\mathbb{E}[x^4] = 3\,\mathbb{E}[x^2] = 3$.)

Step 4. Substitute.

\[L(p) = \tfrac12\Big(p^2\,\tfrac{n+2}{n} - 2p + 1\Big).\]

The minimum and the floor. Set the derivative to zero:

\[\frac{dL}{dp} = p\,\tfrac{n+2}{n} - 1 = 0 \quad\Rightarrow\quad p^* = \frac{n}{n+2}.\]

Plug back in:

\[L(p^*) = \tfrac12\Big(\tfrac{n}{n+2} - \tfrac{2n}{n+2} + 1\Big) = \tfrac12\cdot\tfrac{2}{n+2} = \frac{1}{n+2}.\]

Completing the square gives the form above:

\[L(p) = \frac{n+2}{2n}\,(p - p^*)^2 + \frac{1}{n+2}.\]

Compare this with $\tfrac12(ab - 1)^2$ from the $abx$ model. It is the same loss in the product $p$, with the target moved from 1 to $p^*$, the curvature rescaled by $\tfrac{n+2}{n}$, and a constant floor added.

The derivation used three things about the data. Independence let $\theta$ and $x_q$ drop out, $\mathbb{E}[x^2] = 1$ put the ideal $p$ near 1, and $\mathbb{E}[x^4] = 3$ set how far $p^*$ shrinks below 1 and how high the floor is. For a different input distribution with unit variance, only the fourth moment changes.

Gradient analysis

Since the loss depends on the weights only through $p = vkq$, the chain rule gives $\frac{\partial L}{\partial v} = L’(p)\,kq$, where

\[L'(p) = Ap - 1, \qquad A = \frac{n+2}{n}.\]

The same holds for $k$ and $q$ with the roles swapped, so gradient flow is

\[\dot v = (1 - Ap)\,kq, \qquad \dot k = (1 - Ap)\,vq, \qquad \dot q = (1 - Ap)\,vk.\]

The factor $(1 - Ap)$ is common to all three equations. It is close to 1 when the weights are small and reaches 0 at $p = p^*$. The equations differ in the second factor, which for each weight is the product of the other two.

Observations

We can read a few things off these equations.

  1. A weight only moves when the other two are both nonzero. If all three start small, each of them has a tiny gradient, and the loss sits on a plateau until they have grown enough to pull each other along.

  2. The smallest weights grow fastest. If $v$ is large and $k$, $q$ are small, then $\dot v$ is proportional to $kq$ and is tiny, while $\dot k$ and $\dot q$ each pick up a factor of $v$. So $v$ plays the role of a learning rate for $k$ and $q$.

  3. The differences $v^2 - k^2$ and $k^2 - q^2$ stay constant during training. To see this, multiply each equation by its own weight:

    \[v\,\dot v = k\,\dot k = q\,\dot q = (1 - Ap)\,vkq.\]

    Since $v\dot v = \tfrac12 \frac{d}{dt} v^2$, this says that $v^2$, $k^2$ and $q^2$ all grow at the same rate. One consequence is that weights that start equal stay equal, and with $v = k = q = w$ the dynamics reduce to $\dot w = (1 - Ap)\,w^2$.

  4. How long the plateau lasts depends on how many of the weights start small. Suppose $k$ and $q$ start at size $\varepsilon$ and $v \approx 1$. Early on $\dot k \approx q$ and $\dot q \approx k$, so they grow exponentially and leave the plateau after a time of about $\ln(1/\varepsilon)$. If instead all three start at $\varepsilon$, then $\dot w \approx w^2$, which has solution $w(t) = 1/(1/\varepsilon - t)$, and the plateau lasts about $1/\varepsilon$. If we merge the key and query into one weight $w$, so that $p = vw$, we are back to the $abx$ model with $\dot v = (1 - Ap)\,w$ and $\dot w = (1 - Ap)\,v$, and the plateau is again logarithmic in $1/\varepsilon$.

  5. The only thing we used about attention is that its output depends on a product of scalar weights. Any model like that has the same equations and the same conserved quantities. Deep linear networks have a matrix version: $W_{i+1}^\top W_{i+1} - W_i W_i^\top$ is conserved for adjacent layers, a property called balancedness.

Time to converge from different initializations

The map below shows how long gradient flow takes to converge from each starting point, with $n = 4$. The axes are the initial $k$ and $q$, and the slider sets the initial $v$. We count a run as converged once $p$ is within 5% of $p^*$.

0.60
Time to converge: under 2 2 to 5 5 to 15 15 to 50 50 to 200 over 200 never converges
Hover over the map to read a starting point.

When the product $vkq$ starts positive, no weight has to change sign, and convergence is fast unless the weights start near zero. When it starts negative, one weight has to cross zero first. By the conservation law the three squares shrink at the same rate, so the weight with the smallest magnitude gets there first and flips. The dashed lines are the starting points where the two smallest weights are tied. There both reach zero together, and training stays stuck.