Quick review: run through the questions, block by block

* 3 copies references: the three rows are one and the same list. Whether you write through grid[0], grid[1], or grid[2], you change that one list, so all three rows show it. The right way is [[0] * 3 for _ in range(3)], which builds a new row on every iteration. For how names relate to objects, see two names do not imply two copies.

From：Interviews: the Python you actually use in coding problems

The lambda remembers the variable i, not its value, and looks the value up when it is called; by then the loop has finished and i is 2, so all three return 2. This is late binding. To keep the value from that moment, fix it with a default argument: lambda i=i: i. For a fuller closure example, see defaults and closures.

From：Interviews: the Python you actually use in coding problems

sorted returns a new list from an iterable; list.sort sorts in place and returns None. Both guarantee stability. CPython's sort exploits existing ordered runs: with constant-cost keys and comparisons, worst-case time is O(n log n), while ordered input can be faster. Account separately for expensive keys or long-string comparisons.

From：Interviews: the Python you actually use in coding problems

heapq uses a min-heap by default. Negating numeric values on push and pop implements a max-heap on older versions; Python 3.14 also exposes _max APIs. For the k largest values (1≤k≤n), maintain at most k elements and replace the minimum when a larger value arrives. Scanning costs O(n log(k+1)); sorted output adds O(k log k). See the Top-K cutoff and official API.

From：Interviews: the Python you actually use in coding problems

-7 // 2 = -4 and -7 % 2 = 1. Python's integer division rounds down and guarantees a == (a // b) * b + a % b. C++ and Java round toward zero: -7 / 2 is -3 and the remainder is -1.

From：Interviews: the Python you actually use in coding problems

A list comprehension [...] stores all results. A generator expression (...) computes on demand without building that result list. It still retains execution state and may keep its input alive; once exhausted, it does not restart. A stored list may be preferable when results need several passes. See short-circuiting and iterators.

From：Interviews: the Python you actually use in coding problems

A default value is evaluated once, when the function is defined. The list in def f(acc=[]) is shared by every later call, so whatever one call adds is still there for the next. Use acc=None and if acc is None: acc = [] in the body. A list you pass in is still modified; see defaults and closures for that case and a runnable example.

From：Interviews: the Python you actually use in coding problems

The four common roles are Actor, Critic, Reward, and Reference; they do not require four full models to be resident simultaneously. Actor and Critic update weights in this setup. Reward is frozen after stage 2; Reference is a frozen SFT copy. Actor and Reference start identical, but only the Actor changes as policy training proceeds.

From：RLHF: evaluation and review questions

Preferences learned from limited data may not generalize to later policy outputs. Stronger optimization can find high-scoring answers that people do not actually prefer, such as accommodating a stylistic bias instead of improving the answer.

Reference KL adds a cost for deviating from an anchor. Increasing $\beta$ generally strengthens that cost, but its effect also depends on reward scale. Removing KL does not guarantee collapse, and retaining it does not guarantee freedom from reward hacking. Compare its presence and coefficient with independent evaluation rather than treating a nonzero coefficient as a law. See Reference and Critic for the derivation and numerical example.

From：RLHF: evaluation and review questions

The Critic estimates state value $V_t$ to construct advantages such as $\hat A_t=\hat G_t-V_t$. A suitable baseline can reduce variance; TD or GAE also introduces a bootstrapping bias tradeoff. Training without a Critic is still possible; REINFORCE is one example.

GRPO constructs relative signals from answers to the same prompt instead of training a separate Critic. It saves Critic-related costs but requires group sampling; all-correct or all-wrong groups (see DAPO's dynamic sampling), reward scales, and length normalization still matter. Counting fewer models does not establish lower total training cost.

From：RLHF: evaluation and review questions

DPO uses a derivation: KL-constrained reward maximization has a closed-form optimum, so the reward can be rewritten in terms of the policy and the preference loss differentiated directly. The reward model and the RL loop both disappear.

Standard DPO's cost is offline preference data. The policy changes while the data does not, so it cannot actively explore its current failures. PPO and other online RL methods resample from the current policy and are more natural for exploration, environment interaction, or verifiable multi-step outcomes, but rollout is expensive and training less stable.

DPO therefore cannot replace PPO everywhere, but PPO is not universally better either. Use DPO when high-quality static preferences cover the task; use online methods when the current policy must keep producing new evidence. Online DPO variants reinforce that the real distinction is the data-and-feedback loop, not only the loss name.

From：RLHF: evaluation and review questions

Bradley–Terry directly constrains the gap between chosen and rejected answers to the same prompt. Adding a constant to every score leaves the loss unchanged, so the zero point has no identifiable meaning; a score of $2.4$ is not “2.4 units of satisfaction.”

A fixed Reward Model's raw outputs can of course be used numerically, but comparisons across prompts, domains, or model versions require evidence that calibration and scale are stable. Implementations may also whiten or normalize rewards; that is an optimization choice, not a rule implied by Bradley–Terry.

From：RLHF: evaluation and review questions

A verifier only checks the conditions implemented in it. Incomplete tests, faulty answer extraction, or missing boundary cases can reward incorrect outputs. A deterministic program need not express the full task correctly.

RLVR fits tasks or subgoals with automatically checkable outcomes. Open-ended dialogue can also use format or tool-argument checks, but these do not cover all aspects of answer quality. Reserve independent problems, different test families, and human review where needed to distinguish task learning from checker exploitation. Verifiable rewards starts from a worked counterexample: a constant-output “solution” that passes a single sorting test.

From：RLHF: evaluation and review questions

Four $d_\text{model} \times d_\text{model}$ matrices, so $4d^2$ (without biases) — independent of the number of heads. Splitting into heads only regroups the same parameters.

From：Multi-head attention: from equations to implementation

For batch size B and model width d, time including projections is $O(BTd^2+BT^2d)$; explicitly storing the per-head attention matrices takes $O(BhT^2)$ space. FlashAttention uses tiling to avoid writing those full matrices to GPU memory. It does not remove dense attention's quadratic compute, nor the storage for other activations, KV, and parameters.

From：Multi-head attention: from equations to implementation

They can be shared, but that restricts the matching function. In this self-attention example without extra positional transformations, Q=K makes the raw scores $QQ^\top$ symmetric. Row-wise softmax uses different denominators, so the final weights need not be symmetric. Separate projections let even the raw scores differ by direction; symmetric scores do not imply symmetric attention weights.

$Q=K$ does not make attention fail either. Three cases: if all queries are identical, every row is uniform; if queries are mutually orthogonal with equal norm, the diagonal score is largest, but attention approaches the identity only when the norm is large relative to the softmax temperature; in general, positions more similar to a position itself get more weight, with no guarantee that it attends only to itself.

These internal coordinates are also not unique. For any invertible $R$, setting $Q'=QR$ and $K'=KR^{-\top}$ still gives $Q'K'^\top=QK^\top$: a different internal basis, the same function. Interpreting a single entry of $W_Q$ in isolation is therefore usually meaningless; what matters is the function the projection implements and its causal effect on the output.

From：Multi-head attention: from equations to implementation

Separate projections allow matching features to differ from the content being transmitted. Matching might emphasize entity type while values retain particular attributes. Q/K/V all come from the input, so content already affects scores. Involving V in scoring does not inherently cause collapse; the standard design simply separates these computational roles.

From：Multi-head attention: from equations to implementation

Some heads can be redundant. Head-pruning research removed subsets of heads with little performance change in some models and tasks. That does not imply that most heads can safely be removed from any model: head selection, fine-tuning, and evaluation data all matter.

From：Multi-head attention: from equations to implementation

Adding zero to a blocked logit does not mask it. Setting it to zero still leaves an unnormalized weight of $e^0=1$. Setting it to $-\infty$ before softmax removes it from the denominator. Zeroing weights after softmax and renormalizing is mathematically equivalent on a nonempty allowed support, but zeroing alone is not, and numerical stability is harder to handle.

From：Multi-head attention: from equations to implementation

They make updates relative to the input easier to represent and add an identity term to the Jacobian. Total gradients can still cancel or amplify; the existence of one path is not unconditional stability.

Separate optimization from overfitting by inspecting training error as well as validation error. Higher training error is not explained merely by saying the larger model overfits.

From：Residual connections

Concatenation, as in DenseNet, also preserves information but increases later input widths, requiring parameter and memory management. Addition preserves the current width and makes stacking convenient; it neither permits infinite depth nor requires identical parameter counts in every layer.

Addition also makes "do nothing" a reachable solution ($f = 0$). With concatenation, later layers have to actively learn to ignore what was appended.

From：Residual connections

For one coordinate, write the full expression:

$$\operatorname{Var}(x+f)=\operatorname{Var}(x)+\operatorname{Var}(f)+2\operatorname{Cov}(x,f).$$

Approximately linear growth requires negligible covariance and comparable update variance across layers. If $f=-x$, the output variance is zero instead. Branch scaling controls each increment; final normalization only controls the scale entering the output head, not every intermediate propagation problem.

Inspect activation and gradient distributions before choosing initialization, residual scaling, or normalization.

From：Residual connections

Pre-norm often makes deep networks easier to optimize because normalization stays in the branch rather than the main skip. Suitable learning rates, initialization, and training budgets still matter.

Post-norm normalizes each residual output and has different training and representation behavior. Compare quality at matched budgets rather than claiming one universally wins. The gradient-path derivation is in normalization.

From：Residual connections

On the branch output: $x + \text{Dropout}(f(x))$.

Applying dropout to the skip changes its direct Jacobian to a random mask rather than $I$. This explains the standard branch placement; it does not prove every other dropout design is ineffective.

Embedding, attention, and residual dropout occupy different locations; inspect their settings separately.

From：Residual connections

Both provide additive paths. In $c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t$, holding gates fixed gives the forget gate as the direct derivative along cell state. Values near 1 help propagation across time; the total derivative also includes the gates' dependence on history.

This is a useful analogy between propagation across time and depth, not an equivalence between the architectures.

From：Residual connections

On the logits, before the softmax: $p_i = \text{softmax}(z_i/T)$.

Given the original softmax probabilities, $p_i^{1/T}/\sum_j p_j^{1/T}$ is mathematically equivalent to $\operatorname{softmax}(z/T)$: the common normalization factor cancels. Implementations usually keep logits to avoid probability underflow. Multiplying all probabilities by the same coefficient and renormalizing changes nothing.

From：Decoding strategies: temperature, top-k, top-p

Top-k keeps a fixed $k$ candidates; top-p keeps the smallest set whose cumulative probability just reaches $p$.

The difference is whether the candidate pool changes with the distribution. Top-p can retain few tokens when it is concentrated and many when it is flat. Adaptation does not guarantee quality: a confidently wrong model can still put high probability on the wrong token.

From：Decoding strategies: temperature, top-k, top-p

Model likelihood is not task quality; open-ended generation needs checks for repetition and diversity. Beam search is finite-width approximate search, not a global optimizer. Translation and summarization also admit multiple valid answers. Compare with sampling, reranking, or verification under the same budget.

From：Decoding strategies: temperature, top-k, top-p

With a unique maximum logit, probability concentrates on that token as $T \to 0^+$. Tied maxima share the limiting probability, while greedy implementations typically use a fixed tie-breaking rule.

The formula cannot take $T=0$: it would divide by zero. Some APIs use zero as a shortcut for greedy decoding; others select greedy through a separate option. With fixed logits and a fixed tie-breaking rule, argmax is deterministic. Different kernels or batching can change the computed logits slightly, which can change the winning token when scores are close.

From：Decoding strategies: temperature, top-k, top-p

This article uses temperature → top-k → top-p. Check the framework; interchanging arbitrary steps does not preserve the configuration.

Positive temperature preserves the top-k candidate set: it does not change ranking, provided ties use the same rule.

It does matter for top-p: top-p looks at cumulative probability; temperature changes each token's relative share, so the number of tokens needed to accumulate to $p$ changes. Measured with the same top_p=0.9: at $T=0.7$, temperature first gives 4 candidates and temperature last gives 5; at $T=1.5$ it is 6 versus 5. At $T=1$ the two are equal — so testing only at the default temperature will not reveal it.

From：Decoding strategies: temperature, top-k, top-p

Both act on the logits: $z_i \leftarrow z_i - \alpha_p\mathbb{1}[c_i>0] - \alpha_f c_i$.

With positive coefficients, the presence penalty charges the same amount after 1 occurrence or 10. The frequency penalty charges in proportion to the count. These change relative token probabilities; neither guarantees a change of topic or less repetition of the same idea.

Both also hit tokens that are supposed to repeat — punctuation, pronouns, code indentation and brackets, a person's name that recurs in the text. Turned up too high, the output becomes awkward, and code generation is especially easy to break.

From：Decoding strategies: temperature, top-k, top-p

First identify the actual rollout distribution $\mu$. An untransformed old policy is an easy baseline to audit. Other sampling settings can be deliberate, but the target policy, recorded probabilities, and correction method must be explicit.

In ordinary PPO, the denominator is the old policy that collected the batch, not the current updating policy. Treating samples from $\mu$ as samples from an untransformed old policy changes the estimator. Also check support: actions with zero probability under $\mu$ cannot be recovered from this batch (the same issue in offline estimation is covered in OPE estimators).

From：Decoding strategies: temperature, top-k, top-p

At fixed expert size, total expert parameters grow with N and main expert compute per token grows with k; the router still scores N experts. Mixtral 8x7B duplicates FFNs, not eight complete models, yielding 46.7B total and 12.9B active parameters rather than 8×7B.

From：MoE: review questions

A common router scores experts with a linear layer, then selects top-k. Mixtral renormalizes selected weights; Switch top-1 retains the selected full-softmax probability instead of turning its sole gate into one. Ordinary autograd does not differentiate discrete indices; task gradients flow through continuous gates. Unselected expert parameters receive no task gradient from that token, but their router logits may receive gradients through the full softmax or auxiliary objectives.

From：MoE: review questions

An expert that receives more tokens gets more training and can become even more likely to be chosen; this self-reinforcing imbalance can also leave devices waiting. Switch uses $\alpha N \sum f_i P_i$: assignment fraction $f_i$ is nondifferentiable, while mean router probability $P_i$ is differentiable. Gradients flow through $P_i$. The value is $\alpha$ at perfect balance, not a strict lower bound over all routing distributions.

From：MoE: review questions

Switch top-1 sets capacity from $\frac{\text{tokens}}{N}\times\text{CF}$, rounded according to the implementation. This caps each expert's capacity rather than guaranteeing equal runtime per device. An overflowing expert branch can be skipped while the token continues through the residual path. Top-k and dropless implementations may behave differently; one capacity formula is not universal.

From：MoE: review questions

The bias affects top-k selection while gate weights use original affinities. Overloaded experts have their bias reduced; underloaded experts have it increased. This update needs no auxiliary-loss gradient, but can change the selected set, outputs, and task gradients. V3 still retains a small sequence-wise balance loss ($\alpha=0.0001$), so it is not entirely free of auxiliary objectives.

From：MoE: review questions

Fine-grained experts allow more combinations of smaller experts at comparable expert matmul cost. More combinations do not guarantee better specialization or unchanged communication. Shared experts provide a common computation path; reducing duplication is a design aim, not a predetermined division of knowledge. DeepSeek-V3 uses both; Qwen3-235B-A22B / 30B-A3B have no shared experts.

From：MoE: review questions

Sparse activation does not automatically shrink weight storage by k/N. All parameters must live somewhere: GPU residency, device sharding, and offloading have different costs. Expert parallelism typically adds token dispatch and return traffic, while large batches may use many experts. Comparing memory or speed with a dense baseline requires matching quality, precision, batch, and deployment conditions.

From：MoE: review questions

Both share parameters. ALBERT also factorizes embeddings, so its total reduction cannot be attributed to sharing alone. Looped models keep weights shared and let execution depth vary. Fixed parameters do not mean fixed compute, KV, or latency, or guaranteed depth extrapolation.

From：Looped Transformers: review questions

It adds depth of computation over shared weights, not external facts or independent weights per loop. Whether that computation helps is empirical. Roughly 2 bits per parameter is a small-model synthetic-task measurement, not a measurement of the released Ouro checkpoints or a capacity theorem; more loops need not improve accuracy.

From：Looped Transformers: review questions

Each loop receives input representation $e$ again, as when Huginn combines its prelude output with the current state through an adapter. This avoids relying entirely on the recurrent state to retain the input. It is a design choice, not a necessary or sufficient condition for stable training of every looped model.

From：Looped Transformers: review questions

ACT, PonderNet, exit gates, and depth routers use different objectives. Separate conditional halting probabilities, actual exit probabilities, output selection, and skipped work. The three-step example in adaptive depth averages 2.2 steps. Ouro snapshot 7ea635b executes all configured loops before selecting an output, so selected depth is not executed compute.

From：Looped Transformers: review questions

The weights are shared but hidden-state inputs differ, so full caches usually distinguish (loop, layer). Sharing changes computation and needs error measurements. Checkpoints, prefill policies, and generation lengths differ across papers; protocol differences are not strict reproduction failures. See costs and limits for memory accounting and sources.

From：Looped Transformers: review questions

CoT adds generated positions; looping adds hidden-state updates. They can coexist. Cached per-token forwards do not recompute the whole prefix. Theoretical CoT simulation requires extra structure and input-length conditions; it does not make arbitrary pretrained models interchangeable. Visible text is not necessarily a faithful explanation either.

From：Looped Transformers: review questions

A workflow chains models and tools along code paths written in advance; an agent lets the model decide the process and the tools. When the steps can be listed in advance, when there is no verification signal, or when a wrong step is costly (sending, paying, deleting), do not reach for an agent directly: write a workflow first, or add a person's confirmation at the critical steps.

From：Agents: review questions

A model (decides the next step), tools (act on the world), a loop (results go back to the model), and a stopping condition (done, or out of budget), plus a context or memory holding intermediate state. ReAct's "think a step, take a step" is the most common form of the loop.

From：Agents: review questions

Because its verification signal is the strongest: code runs, tests pass, every step gets objective feedback, and the agent can try again in a sandbox, with a person still reviewing the diff once tests pass. Settings such as customer support or personal assistants are much harder to judge automatically and lean more on rules, permissions, and people. Even in coding, passing tests does not mean the change is right.

From：Agents: review questions

Not by single-run success alone: check whether the same task succeeds in every one of several independent repeats, as with τ-bench's pass^k (the probability that all k runs succeed). Also check policy compliance, whether tool calls are correct, whether hard cases are handed to a person, and read complete run traces rather than only the final answer.

From：Agents: review questions

Capability first: if an open model misses the bar on your own evaluation set, use a frontier model. If data has strict boundaries, first check whether an API's region, retention, contracts, and private connections satisfy them; self-host if they cannot. With high, steady traffic, spare compute, or deep customization needs, self-hosting may be cheaper and more controllable once idle capacity, operations, and maintenance are counted; there is no universal volume threshold. At low or uncertain volume, start on an API and measure quality and cost. In practice they are often mixed: routing, cascades, distillation, fallback.

From：Agents: review questions

If steps are independent and any failure is unrecoverable, errors compound: at 95% per step, all 20 steps are right only about 36% of the time; at 99% per step, about 82%. Real errors correlate and systems can retry or recover, so the product is a warning that motivates end-to-end evaluation and recording where failures happen, not a prediction.

From：Agents: review questions

A model optimizes the reward you actually give it, not the goal you had in mind, and any gap between the two gets exploited. In OpenAI's boat race the reward was the pickups along the course, so the agent spun in a circle farming them and outscored finishing the race. LLMs do the same: reward "the tests pass" and an agent may edit the tests; in RLHF, if the reward model favors long or agreeable answers, the policy learns to write longer and flatter more.

From：Agents: review questions

A cascade lets the small model answer first and decides whether to escalate after seeing the answer, so it judges better — but escalated requests are paid for twice, and wait for two models in series. A router decides before any answer exists: each request pays for one model call plus the router and waits less, but it can misroute; catching that takes an output check or a fallback, whose cost and delay count too. Either one only pays off when difficulty is judged accurately.

From：Agents: review questions