Mechanistic interpretability: review questions

Behavioural evaluation asks "is the output right on these inputs?"; interpretability asks "which internal quantity produced this output, and does changing it change the output as predicted?". Two models can both answer "capital of the state containing Dallas" correctly, one by first thinking of Texas and then reaching Austin, the other by having memorised the pairing; Claude 3.5 Haiku has both routes.

It is worth using when the same score with different mechanisms affects out-of-distribution behaviour, when a model may change its behaviour because it knows it is being tested (the Sonnet 4.5 system card), and when you need hypotheses without labels. The cost is that it is slow, expensive and narrow, and often explains a replacement model. See the guide.

It only shows that the information can be read linearly from that layer. It does not show that the model's output uses it, nor rule out that the accuracy comes from the probe's own capacity to fit. Hewitt & Liang's control tasks give each word type a random but fixed label: for part-of-speech tagging on ELMo's first layer, a linear probe gets 97.2 against 71.2 on the control task, a selectivity of 26.0, while an MLP probe gets 97.3 against 92.8, a selectivity of only 4.5.

To make it solid: report selectivity; use random directions or randomly initialised models as baselines (Anthropic's sleeper agent probe reached AUROC 99.3%, and the best of 1,000 random directions still reached 95.3%); intervene along the probe direction and check that the output changes as expected; and keep in mind that subspace patching can create illusions (Makelov et al.). See Features and probes.

Superposition is a network representing more features than it has dimensions, by letting feature directions be not quite orthogonal and tolerating some interference. The toy model of Elhage et al. is $x'=\mathrm{ReLU}(W^\top Wx+b)$: with dense features it learns only the $m$ most important ones; as features get sparser superposition appears, and it does so as a phase change.

By hand: put 3 features 120° apart in 2 dimensions, so each pair interferes at −0.5. With one feature active, the negative interference is cut off by the ReLU and reconstruction is exact; with two active, each loses half; with all three active they cancel completely. If each feature is active with probability $p$, the expected loss is $1.5p^2+1.5p^3$, against $p$ for dropping one feature: at $p=0.1$ that is 0.0165 against 0.1, and the break-even point is about 0.457. Without the ReLU, superposition no longer pays. See Features and probes.

$f=\mathrm{ReLU}(W_{\rm enc}x+b)$, $\hat x=W_{\rm dec}f+b'$, and the loss is reconstruction error plus $\lambda\sum_i f_i\|W_{\rm dec}^{(i)}\|$; multiplying by the decoder column norm stops the model from dodging the penalty by shrinking $f_i$ and enlarging the decoder vector. L1 causes shrinkage: non-zero activations are systematically underestimated. TopK (OpenAI) keeps only the $k$ largest per token, setting sparsity directly and leaving only the reconstruction term; JumpReLU (Google DeepMind) learns a threshold per latent and penalises L0 directly, trained with straight-through estimators.

By hand: $x=(1,1)$, with dictionary directions $(1,0)$, $(0,1)$ and $(0.707,0.707)$. L1 prefers using only the diagonal direction (a cost of $\sqrt2$ against 2); at $\lambda=1$ the coefficient is 0.914 and only 87.5% is reconstructed; for $\lambda\ge2\sqrt2$ nothing fires. TopK with $k=1$ gives a coefficient of $\sqrt2$ with no shrinkage. See Sparse autoencoders.

Dead latents never fire during training, so they get no gradient. Anthropic's 1M, 4M and 34M SAEs had roughly 2%, 35% and 65% dead; OpenAI, initialising the encoder as the transpose of the decoder and adding the AuxK auxiliary loss, had only 7% dead at 16 million wide (up to 90% without mitigations). Feature splitting: as the dictionary grows, one feature splits into finer ones, such as "San Francisco" going from 1 to 2 to 11 features. Absorption: when features form a hierarchy, a parent feature fails to fire where it should because child features absorb it, and changing size or sparsity does not fix it.

Changing the seed: 131K-wide SAEs on an MLP of Llama 3 8B share only about 30% of their features, and TopK depends more on the seed. So an SAE gives a pragmatically useful decomposition, not a complete list of the model's features. See Sparse autoencoders.

Their 2025-03-26 progress update: on detecting harmful intent with Gemma 2 9B IT, tested out of distribution, a linear probe on the residual stream reached OOD AUROC 0.999 and SAE probes were distinctly worse. Around the same time, Kantamneni et al. could not consistently beat baselines in four difficult sparse-probing settings; AxBench found prompting best for steering and difference-in-means best for detection, with SAEs competitive on neither; SAEBench found proxy metrics disconnected from practical usefulness. In December 2025 the team went further and argued for trying prompting, steering and probing first.

SAEs still suit exploration when you do not know what to look for: in the auditing game, one person found the hidden objective in 90 minutes through an SAE feature (the paper notes semantic search could reproduce it); OpenAI compared models before and after finetuning with SAEs and found a "toxic persona" feature; and they help find spurious correlations in training data. When you know the concept and have labels, start with a probe or difference-in-means. See Sparse autoencoders.

Denoising puts clean values into the corrupted run and asks "is this enough to rescue the behaviour?"; noising puts corrupted values into the clean run and asks "can the model do without it?". With backup components the two disagree: in the 3-node example, denoising B restores 86% but noising B loses only 43%, because C steps in when B is absent. Path effects are not additive either; the two paths sum to 143%.

The IOI circuit recovers 87% of the full model's logit difference (faithfulness), but faithfulness cannot see backups: when the name movers are removed, the backup name movers take over. Completeness requires that removing the same set of components from the circuit and from the full model has similar effects; in the example, removing B leaves the model at 2.0 and the circuit at 0, exposing that C was left out. See Circuits.

First train a cross-layer transcoder to replace every MLP; for a single prompt, freeze the attention patterns and normalisation and add error nodes so that the local replacement model matches the original's outputs; edge weights are $A_{s\to t}=a_s w_{s\to t}$; prune, group, and go back to the original model to validate with interventions. The authors list the limits themselves: it does not explain why attention looks where it does (QK attributions came later), error nodes can hide the key computation, it looks at single prompts, and the replacement model may not share the original's mechanism; it is satisfying on about a quarter of prompts.

What to ask: was the problem implanted, or did it occur naturally? Who reported it, and has a third party reproduced it? What was it compared with (random directions, chain-of-thought monitoring, behavioural evaluation)? What is the false-positive rate? For example, Apollo's deception probes reach AUROC 0.96–0.999, yet the authors say this is not enough for a robust defence; inhibiting evaluation awareness in Sonnet 4.5 raised misaligned actions to at most 8–9%, self-reported by the lab and varying by dataset. See Circuits.