Run a four-layer block three times: that is twelve layer evaluations with only four layers of stored weights. Each pass receives the state produced by the previous one, so the result usually changes. A looped model reuses parameters to spend more computation; it does not acquire new facts merely by looping.
Depth and parameters normally come as a pair
In an ordinary Transformer without parameter sharing, each layer has its own weights. Adding layers in that design increases both depth and parameters. Recurrence is one way to separate these budgets. Generating intermediate tokens or using tools can also add problem-solving computation, but those are different mechanisms.
Loop the block
A looped Transformer stores a single $k$-layer block and applies it $L$ times:
$$h^{(t+1)} = f_\theta\big(h^{(t)},\, e\big), \qquad t = 0, 1, \ldots, L-1$$
$\theta$ is shared across loops; $e$ is an input representation that some designs re-inject at each step. The recurrent block stores $k$ layers of weights and performs $kL$ layer evaluations. Embeddings, output heads, and extra adapters count separately. This is not equivalent to having $kL$ independently parameterized layers.
Shared parameters need not produce identical outputs. Starting at 0, repeatedly applying $h\leftarrow 0.5h+1$ gives 1, 1.5, and 1.75: the input state changes. In contrast, $h\leftarrow 2h$ starting at 1 gives 2, 4, 8 and grows without bound; starting at 0, it stays at 0. Reuse alone guarantees neither convergence nor improving answers.
An old idea, used in a new way
- Universal Transformer (2018) shares transitions across positions and depth and studies ACT for allocating computation.
- ALBERT (2019) combines layer sharing with factorized embeddings to reduce parameters. Sharing does not automatically reduce executed layers; the 108M → 12M comparison is not due to sharing alone, nor does it establish unchanged runtime.
- Huginn and Ouro, introduced in 2025, apply recurrence to language-model pretraining. Depth is an experimental variable, not a universal reasoning switch: extrapolation beyond training depends on the checkpoint and task.
How this series reads
- Why loops help reasoning: serial steps, the theory, and how it relates to CoT (interactive)
- How many loops per token: ACT, PonderNet, Ouro's exit gate, Mixture-of-Recursions (interactive)
- What today's looped LMs look like: Huginn, Ouro, Relaxed Recursive Transformers
- Costs and limits: latency, KV cache, training stability, and the scope of knowledge-capacity experiments
- Review questions: interview questions and a self-check