Interview basics: most of them ask the same thing

$$\mathcal{L}_{\text{MSE}} = (p-y)^2 \qquad \mathcal{L}_{\text{BCE}} = -\big[y\log p + (1-y)\log(1-p)\big]$$

This question is about the gradient, not about reciting formulas. Key fact: $\sigma'(z) = p(1-p)$.

$$\frac{\partial \mathcal{L}_{\text{MSE}}}{\partial z} = 2(p-y)\cdot p(1-p) \qquad\qquad \frac{\partial \mathcal{L}_{\text{BCE}}}{\partial z} = p - y$$

For BCE the $\sigma'$ cancels:

$$\frac{\partial \mathcal{L}_{\text{BCE}}}{\partial p} = \frac{p-y}{p(1-p)} \;\Longrightarrow\; \frac{\partial \mathcal{L}}{\partial z} = \frac{p-y}{p(1-p)}\cdot p(1-p) = p-y$$

The consequence: when $y=1$ and the model is confidently wrong ($z\to-\infty$, $p\to0$) —

• MSE gradient $\approx 2(0-1)\cdot 0\cdot 1 = 0$. The gradient vanishes exactly when the error is largest.
• BCE gradient $= -1$. Maximum gradient exactly when maximally wrong.

One layer deeper: for logistic regression BCE is convex in the weights; squared error with a sigmoid isn't.

Separate optimization from calibration: squared error on binary probabilities corresponds to the Brier score, which is also a proper scoring rule. Both it and BCE favor the true conditional probability under the ideal population objective; finite data and restricted models do not guarantee calibration. The comparison here concerns gradients, not whether either loss is allowed to estimate probabilities.

In practice: always binary_cross_entropy_with_logits. Computing $p$ then taking its log underflows to $-\infty$. The stable form:

$$\mathcal{L} = \max(z,0) - zy + \log\big(1+e^{-|z|}\big)$$

Say first: one reason vanilla RNNs struggle with long-range relationships is repeated gradient propagation through time.

$$\frac{\partial h_t}{\partial h_{t-k}} = J_t J_{t-1}\cdots J_{t-k+1},\qquad J_t=\operatorname{diag}\big(\tanh'(a_t)\big)W_h$$

This is the state-to-state Jacobian; backpropagation uses its transpose. The weights are shared, but activations change, so the Jacobians need not be identical. A common bound $\|J_t\|_2\le c<1$ guarantees decay along this path. A norm greater than one at one step does not guarantee explosion: directions and the remaining product matter too.

Clipping limits an update; it cannot recover a signal that has already vanished, or replace diagnosing the source of instability.

The fix: a cell state with an additive update.

$$c_t = f_t \odot c_{t-1} + i_t \odot g_t, \qquad h_t = o_t \odot \tanh(c_t)$$

Holding the gates fixed, the direct cell-state path has derivative $\partial c_t/\partial c_{t-1}=\operatorname{diag}(f_t)$. This path only rescales elements, making long-range signal easier to preserve when the forget gate is near one. The total derivative also includes the gates' dependence on previous states; LSTMs do not eliminate vanishing gradients.

One line: LSTM replaces "multiply by a matrix at every step" with "gated additive accumulation."

Detail: sigmoid bounds a gate between zero and one; tanh allows signed candidate content. These choices fit the roles of gates and values and also affect optimization. Saturation has not disappeared.

Two connections that lift the answer:

• Cell states and residual streams are a useful analogy, not identical structures. Both offer additive paths; an LSTM gates memory through time, while residual connections typically preserve an identity term through depth.
• Transformers offer several advantages. They process positions in parallel during training, shorten paths between visible positions, and provide content-based access to history. LSTMs mitigate rather than solve long-range learning problems. Autoregressive Transformer generation still proceeds token by token.

Why normalize at all: it helps control the scale seen by sublayers and often makes deep networks easier to train. It works alongside initialization, residual scaling, and the optimizer; networks without normalization are not universally untrainable.

Why do Transformers commonly use LayerNorm rather than BatchNorm?

1. Padding affects BN statistics when it is included in the reduction without being excluded;
2. BN depends on batch size and composition and defaults to running statistics during inference. Autoregressive decoding differs from training, so this requires care;
3. Including the time axis in training BN statistics lets future tokens affect earlier positions. A causal attention mask does not prevent that leak;
4. Token-wise LN over features mixes neither batch examples nor time positions and uses the same statistics rule in training and inference. That does not make every other part of the model identical across modes.

Where it goes — be able to write both:

Post-LN (original, 2017)      Pre-LN (modern)
x = LN(x + Attn(x))          x = x + Attn(LN(x))
x = LN(x + FFN(x))           x = x + FFN(LN(x))
                             ...
                             x = LN(x)   ← a common final LN

Why Pre-LN is common: it preserves an identity term along the residual trunk, so gradients need not pass through LN at every layer and training relies less on warmup; that is not a guarantee that every model can drop warmup. The Jacobian derivation, and how far Xiong et al.'s result carries, are in question 3 of normalization.

Implementation check: Pre-LN models commonly normalize the final residual output before the output head. This is a common recipe, not a mathematical necessity. Match the particular model when reproducing it.

One deeper: RMSNorm omits mean-centering and usually additive bias, rescaling by RMS with a learned gain. Its success shows that some models train well without centering, not that centering is universally irrelevant. See the LayerNorm analysis and RMSNorm paper.

Say first: it's what makes training all $T$ positions in one forward pass equivalent to training them one at a time.

The autoregressive objective is $\prod_t p(x_t\mid x_{<t})$. In a common implementation, position $t$ receives $x_t$ and predicts $x_{t+1}$. Seeing itself is allowed; seeing the future token $x_{t+1}$ leaks the target. The causal mask allows the current and earlier positions, matching the information available during generation.

So the mask isn't there to make the model stronger. It's there so training and inference are the same model. Without it you'd need $T$ separate forward passes.

It goes at step ② of the skeleton: after the scaled dot product, before the softmax.

mask   = np.triu(np.ones((T, T), dtype=bool), k=1)   # strictly upper = blocked
scores = np.where(mask, -np.inf, scores)             # add -inf, don't zero

Why it must be before the softmax (the follow-up): adding $-\infty$ makes $e^{-\infty}=0$, and the softmax renormalizes over the remaining positions — mathematically identical to those positions not existing.

Zeroing after the softmax breaks normalization — rows no longer sum to one, and unevenly: position 1 can only see itself, loses the most mass, and gets scaled down hardest. You've multiplied each position by an arbitrary, meaningless attenuation. Multi-head attention works it through with two scores.

Engineering detail: a fully masked row has no valid probability distribution. A plain softmax over all -inf produces NaNs; finite negative values may produce a uniform distribution, which is not a valid fix. -1e9 also exceeds fp16's finite range. Explicitly handle queries without valid keys, check your attention API's all-masked behavior, and exclude padding from the loss. Masking after softmax and correctly renormalizing can be mathematically equivalent; merely zeroing entries is not.

Why BERT doesn't need it: it isn't autoregressive. The objective is MLM, and bidirectional visibility is the design, not a leak.

Yes, substantially. Start by separating two things people conflate:

Convolution gives translation equivariance, not invariance:

$$f(T_x(I)) = T_x\big(f(I)\big)$$

Shift the input and the feature map shifts by the same amount. Invariance — output unchanged — comes from pooling afterwards, and it's approximate and local.

Rotation is neither. The kernel has a fixed orientation, so a 45-degree edge and a 135-degree edge activate entirely different filters. Nothing in the architecture makes them the same object.

Why the asymmetry is structural: translation equivariance falls out of weight sharing plus locality — the operator gives it to you free. Rotation equivariance isn't in the operator, so you have exactly two options:

1. Buy it with data: rotation augmentation. The network learns redundant filters, one set per orientation. You're spending model capacity on invariance, and only over the range you augmented.
2. Change the operator: group-equivariant CNNs, steerable CNNs, harmonic networks — or a spatial transformer that learns to canonicalize the pose.

One line: translation equivariance comes free with convolution; rotation equivariance or invariance costs, and you pay in either data or the operator.

To make the conversation interesting: even translation invariance is weaker than assumed — strided downsampling aliases, so a one-pixel shift can flip the prediction. The fix is anti-aliased downsampling.

Quick memory

Most Transformer fundamentals test four things: shape closure, causal information flow, gradient flow, and train/inference equivalence.

Interview answer

I first write the input and output shapes and identify the reduction axis. Then I check causal masks and data boundaries for leakage, inspect residual paths and normalization for gradient flow, and finally verify that training, prefill, and decode implement equivalent computations.