Looped Transformers: what today's looped LMs look like

Huginn: prelude, recurrent block, coda

Huginn (Geiping et al., 2025) is a 3.5B model in three parts:

• prelude (2 layers): turns the input into an embedding $e$;
• recurrent block (4 layers): runs over and over; every loop an adapter concatenates the current state $s$ with $e$ and projects it back to the original width, so the input is re-injected each loop; the starting state $s_0$ is random;
• coda (2 layers): turns the final state back into a distribution over the next token.

Training samples recurrent depth with a configured expectation around 32 and backpropagates through at most the last eight loops. Earlier states still affect the forward computation, but the full gradient chain is not retained; this is not an unbiased gradient of full unrolling. The paper reports test-time compute benefits after about 800B training tokens. A “50B-equivalent” compute budget is not 50B parameters of capacity or monotonic improvement on every question.

Ouro: looping built into pretraining

Ouro (2025; details from v5, 2026-07-01; the cited content is unchanged in v6, 2026-10-11) repeats the decoder stack, not the token-embedding lookup:

The 7.7T-token training lineage includes the shared phase before the sizes branch; it is not 7.7T independent tokens at 2.6B from initialization. Table 1 and §4.3 specify eight loops for the shared first 3T tokens, then four for stability as the 2.6B branch is created. Four is the final configuration, not the entire training history. Depth extrapolation is not guaranteed to help. See adaptive depth for the distinction between an exit gate and actually skipping computation.

Relaxed Recursive Transformers: converting an existing model

Bae et al. initialize shared layers from a pretrained model and continue training. Depth-specific LoRAs allow variation and add parameters. Reported comparisons use specific tasks and training budgets, not lossless compression by tying weights alone. The 2–3× throughput potential comes from theoretical / simulated analysis with oracle early exits, not a guaranteed deployment speedup.

Side by side

See adaptive depth for Mixture-of-Recursions routing and cache constraints. In comparisons, fix checkpoint, maximum depth, input / output lengths, dtype, KV policy, and batch size. Fewer parameters and faster serving are different claims.