Interview questions
How does a looped Transformer differ from ALBERT's parameter sharing?
Both share parameters. ALBERT also factorizes embeddings, so its total reduction cannot be attributed to sharing alone. Looped models keep weights shared and let execution depth vary. Fixed parameters do not mean fixed compute, KV, or latency, or guaranteed depth extrapolation.
What does looping add, and what does it not add?
It adds depth of computation over shared weights, not external facts or independent weights per loop. Whether that computation helps is empirical. Roughly 2 bits per parameter is a small-model synthetic-task measurement, not a measurement of the released Ouro checkpoints or a capacity theorem; more loops need not improve accuracy.
What is input injection, and why is it needed?
Each loop receives input representation $e$ again, as when Huginn combines its prelude output with the current state through an adapter. This avoids relying entirely on the recurrent state to retain the input. It is a design choice, not a necessary or sufficient condition for stable training of every looped model.
What are the approaches to adaptive depth? Does Ouro's early exit save compute in the released code?
ACT, PonderNet, exit gates, and depth routers use different objectives. Separate conditional halting probabilities, actual exit probabilities, output selection, and skipped work. The three-step example in adaptive depth averages 2.2 steps. Ouro snapshot 7ea635b executes all configured loops before selecting an output, so selected depth is not executed compute.
Why does a looped model's KV cache grow? Can the loops share it?
The weights are shared but hidden-state inputs differ, so full caches usually distinguish (loop, layer). Sharing changes computation and needs error measurements. Checkpoints, prefill policies, and generation lengths differ across papers; protocol differences are not strict reproduction failures. See costs and limits for memory accounting and sources.
How does looping relate to chain-of-thought?
CoT adds generated positions; looping adds hidden-state updates. They can coexist. Cached per-token forwards do not recompute the whole prefix. Theoretical CoT simulation requires extra structure and input-length conditions; it does not make arbitrary pretrained models interchangeable. Visible text is not necessarily a faithful explanation either.
Self-check
- why an ordinary Transformer's depth and parameters are tied, and how looping unties them;
- which budgets differ between shared and independent weights when both execute twelve layer evaluations;
- how ACT's accumulation, PonderNet's stepwise sampling, and Ouro's cumulative-probability threshold differ, and whether all require per-token decisions;
- when full KV storage scales directly with loop count, and why Huginn's 2+4+2 layout needs only 2.5 times the KV at four loops versus one;
- which tasks suit a looped model and which do not.
Papers
- Universal Transformers: sharing across steps, and ACT
- ALBERT: cross-layer parameter sharing (for contrast)
- PonderNet: probabilistic halting
- Looped Transformers as Programmable Computers
- Looped Transformers are Better at Learning Learning Algorithms
- Reasoning with Latent Thoughts: On the Power of Looped Transformers
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach: Huginn
- Relaxed Recursive Transformers
- Mixture-of-Recursions
- Scaling Latent Reasoning via Looped Language Models: Ouro
- Continuous Depth Batching: KV-sharing evaluation in v2, Appendix E.1
- MELT: training-free KV-sharing evaluation in v3, Appendix B