Content coverage: what is here, and what is missing?

All seven source sections and the main equations were read in Chrome, confirming Kimi K3. New coverage includes order swaps, causal masks, matrix updates, ordered products, numerical limits, and cache tests. The two original reports distinguish mechanisms, ablations, and interpretation. Model training and kernel performance were not reproduced.

Can a model track order without positional encoding? · Gated DeltaNet: decay the state, then correct its prediction

Kimi K3 LatentMoE reference prose and equations read. Primary reports support three widths, projection costs, full-width router input, SiTU/RMSNorm, and next-step quantile balancing. Original CPU examples are not training or communication benchmarks.

MoE: fine-grained and shared experts · LatentMoE: must experts be as wide as the backbone?

Source prose and derivation read. Adds scaling versus shifting, explicit independence assumptions, a correlated counterexample, and attention versus CE gradients, checked by enumeration and autograd.

Multi-head attention: from equations to implementation

Reference prose, architecture diagram, and attention equation were read. Original papers and code support inputs, post-LN blocks, three masks, MLM/NSP, task heads, and retrieval boundaries. Original examples and loss checks are not a pretraining reproduction.

BERT: how an encoder learns to read

Source prose, architecture figure, and softmax equation read. Distinguishes router and expert weights, deterministic top-k and sampling, and the scope of sparse FFNs. Existing dispatch and gradient tests remain; no universal speed or fine-tuning claim is adopted.

MoE: how the router picks experts · MoE: load balancing

Source prose and equations read. Adds actual-token probabilities, weighted NLL, overlapping-window targets, and post-shift counting; original CPU examples are not model evaluations.

BLEU, ROUGE, edit distance and perplexity: what does the score compare?

Source prose, figure, and equations read. Adds position-dependent projection algebra, a 2D rotation counterexample, one shared softmax, combined-width scaling, and expanded/latent output and gradient checks. The limitation concerns fixed projection absorption, not RoPE validity.

MLA and sparse attention: storing less or reading less?

Read six concept groups and the copy examples through the comments, without running or republishing source code. Original examples cover identity, mutability, arguments, any/all, state organization, and copying, with rebinding, memo, iterator consumption, and default-value boundaries checked against Python documentation.

Interviews: the Python you actually use in coding problems · Python: names, copies, and function calls

This is an index into the algorithm route, not a missing problem. The reference index has been read and six method pairs added alongside our existing notes; we do not reproduce its full problem list.

LeetCode: learn the pattern, not just the answer

This index maps to multiple PyTorch and foundations notes. All 27 reference articles have been read; unused source-image equations and reference code are not claimed as verified.

PyTorch: from equations to training code

The source's grouped-routing explanation and code were read to the end, not executed or republished as an interview question. Original dispatch, expert computation, and weighted accumulation match a dense oracle and gradient checks. Group scoring, insufficient candidates, and negative logits are covered. CPU teaching code does not establish distributed performance.

MoE: how the router picks experts · MoE: load balancing

All three sections and code read, with stable/online equations inspected. Adds Algorithms 2/3 scan counts, distinguishing memory accesses from CPU/GPU transfers; full probabilities still need an output stage. No source-code execution or GPU benchmark.

Seven ML modules to implement by hand · FlashAttention and PagedAttention: what does each save?

Source prose and equations were read. Coverage adds the KT denominator, a 1.8 versus 1.0 example, probability gradients, padding, and packing scope. Hard-count squared error alone cannot train a router; V3 affinity statistics and bias-adjusted dispatch are distinguished. CPU checks are not a training reproduction.

MoE: load balancing

Prose and six summary points read; not every figure audited. Site comparisons fix width, total parameters, or compute separately, without assuming equal memory or prescribed expert domains. System costs are already covered.

MoE: why make a model sparse · MoE: what it costs to train and serve

Read Qwen through Qwen3-2507 and the summary. Official launch pages supply the early timeline instead of the reference dates or blanket KV savings. Checkpoints, templates, training and later variants remain separate; family benchmarks were not rerun.

Qwen 1–2.5: separating architecture changes from training changes · Qwen deep dive: one family across scales and reasoning modes

Definitions, trial division, factor-pair proof and both code blocks read to the end; reference code not run. Original coverage includes inclusive/exclusive bounds, isqrt, sieve invariants, segmented starts and factorization; iteration counts are not prime counts.

Primality and sieves: one number or a whole range?

Prose, two equations, and BCE code read; reference code not executed. Original explanations and CPU tests cover stable CE/BCE, soft targets, masks, reductions, and gradients.

Cross-entropy: from one prediction to a training loss

Read the prose and formulas in Chrome. Checked textbook §3.13, separating surprisal from distribution entropy; added a rare-event example, CE decomposition, and changed-target comparison. The reference's event-entropy wording is not adopted.

Distillation: what is the student learning from the teacher? · Three KL estimators: similar values, different gradients

All 52 lines read, including the virtualized ending; source code not run or copied. Checked Q/K/V, mask conventions, scaling, dropout and output projection; corrected site snippet precision/input checks to match its full script.

Multi-head attention: from equations to implementation · Seven ML modules to implement by hand

All 82 lines read. The source mixes local seq with global seq_length; the site does not. Added successive variable-length comparisons and softmax boundary tests for both language snippets, beyond a single output-shape check.

Multi-head attention: from equations to implementation

All 34 lines of adjacent-pair code and its introduction read; source code not run. Site explains split-half weight permutations, actual positions and caching, with rotation/cache numerical checks rather than a universal checkpoint convention.

The Transformer architecture · RoPE and long context: fitting the input is not understanding it

Full prose, bit/exponent descriptions and summary read. Site rounding/update examples separate range from precision; no blanket truncation-only conversion, identical range or faster-runtime claims. CUDA performance not tested.

Precision and memory: why can BF16 training still run out of memory?

All four sections read; inspected the standardization formula, LN-axis figure and table, without auditing every BN diagram cell. PyTorch 2.8 grounds standard deviation, N/L axes, normalized_shape, running statistics and train/eval, separate from input preprocessing.

Normalization: BatchNorm, LayerNorm, and RMSNorm

Full prose and both diagrams read. Corrected the claim that a padding mask adds tokens; clarified the strict upper triangle, visible diagonal, combined padding/causal rules, and cross-attention scope.

Vanilla Transformer · Multi-head attention: from equations to implementation · BERT: how an encoder learns to read

Prose and BERT/ERNIE/autoregressive diagrams read. Direct comparisons cover visibility, parallel training versus incremental generation, MLM/next-token objectives, and 2019 phrase/entity masking without equating architecture with objective.

BERT: how an encoder learns to read · Decoder-only: autoregressive generation

All four sections read. ULMFiT, ELMo, BERT, and Transformer papers correct claims about absent LSTM pretraining or architecture-only generalization judgments; computation paths, task examples, and length limits are covered.

RNN and LSTM · BERT: how an encoder learns to read

Adds shared byte boundaries, proportional allocation, and path-versus-text probability examples; checks the June 2026 method and October 2026 analysis. Original CPU checks, not training replication.

Distillation: what is the student learning from the teacher?

Prose re-read; adds group-variation probability, 3-step GAE, fragment bootstrap and weighting examples, checked against the official GLM-5.2 description. Unequal lengths do not invalidate GRPO; critics do not prove causal credit. No model training performed.

The LLM bridge: which RL intuitions transfer?

Prose and DFT discussion read. Added checkpoint, reward-validity, and equal-budget experiments. DFT detach gradients are verified against the paper, without a universal loss threshold.

SFT: how a conversation becomes a training signal

Four source sections, code, and equations read in Chrome; source code not run. Adds scalar recomputation, selective retention, and a full softmax-backward counterexample. Forward values, non-leaf gradients, and disk recovery remain distinct; FlashAttention still retains state. CPU checks are not GPU memory measurements.

Precision and memory: why can BF16 training still run out of memory? · Multi-head attention: from equations to implementation · A saved model is not always a resumable run

Reference prose and formulas read. Separates prefix sources, full-distribution/sample supervision, fixed-prefix and trajectory gradients; qualifies reverse-KL claims. No full training reproduction.

Distillation: what is the student learning from the teacher?

Prose read. Added position-wise shifting/masking, context gradients, tool messages, and EOS/PAD distinctions. Assistant-only supervision is a convention, not a universal rule.

SFT: how a conversation becomes a training signal

Prose and trajectory formula read. Rewrites completion/turn/episode/fragment boundaries, record fields, tool masks, termination/truncation, behavior probabilities, and asynchronous budgets. Examples are not GPU benchmarks.

Post-training infrastructure: sampling, numerics, context · RL experiments: correctness before speed

Reference prose and key formulas read. Runnable examples separate zero loss from nonzero gradients, group centering, length reduction, local batches, and the Reference. Full model training is not reproduced.

PPO, GRPO, and DPO: turning feedback into an update

Read the reference prose and all three formulas. Explicit direction, sampling, and support accompany a localized sample diagram, variance counterexample, and three direct-loss gradient checks. DeepSeekMath and a 2025 gradient analysis ground the distinction; no blanket trainer substitution.

Three KL estimators: similar values, different gradients

Read the source, update equation, and PPO/DPO discussion. PyTorch 2.8 clarifies optional SGD momentum, gradient-before-moments ordering, and a 100-parameter state comparison. Existing dtype, LoRA, lazy-state, and sharding ledgers remain; no universal training/inference multiplier.

Precision and memory: why can BF16 training still run out of memory?

Reference read to the end. Added request tracing, oracle comparisons, target checks, serving mismatches, and regression tests. Original teaching fixtures, not model experiments; no blanket full-tuning or capacity claims.

Compare experiments and replace models safely · Applying judges to RAG, agents, and memory

Read the prose to its ending and inspected both hierarchy figures. Official CUDA documentation separates SM computation from on-chip storage; fixed capacities and computing-in-SRAM shorthand are not adopted. Existing host/device flow, async-safety and pipeline examples remain; no CUDA performance experiment was run.

From batches to a training loop: running is not learning · FlashAttention and PagedAttention: what does each save?

Read the prose, three update equations, 14-line example, and distributed ending; reference code was not executed. Moonlight v1 §2.3 informs an original shard counterexample and the distinction between collectives, orthogonalization, momentum shards, and gathering. No universal half-memory, exact-orthogonality, or zero-overhead claim; no multi-GPU training run.

Muon: updating a matrix · From SGD to AdamW: What Changes in One Update?

Reference prose read to the end. Original calculations and tests cover the envelope, conditional distributions, best-of-N, prompt coverage, and training records; no model-training replication.

Rejection sampling: generate more, then keep what?

Read the prose, advantage equation, and role diagram. The original paper distinguishes multiple responses from one policy, Reference conditional distributions rather than training data, and outer-loop refreshes. Existing four-response arithmetic and cost tradeoffs remain; a reward model is not mandatory for every GRPO setup.

GRPO, DPO, RLVR: which part does each change? · PPO, GRPO, and DPO: turning feedback into an update · The LLM bridge: which RL intuitions transfer?

After Chrome access recovered, all five sections and the main objective equations were read through the ending. Primary papers and original gradient checks distinguish Dual-clip, CISPO, and SAPO. The reference's CISPO misspelling, fixed-default caps, hard update-bound claims and exclusive GSPO adoption claim are not adopted. No model training was reproduced.

PPO clipping: why not keep increasing the probability? · What exactly is being clipped? · DAPO: how sampling, length, and clipping change training · GSPO and ASPO: how should one answer share its gradient?

Reference prose, clamp code, and formulas read. Original autograd examples check four sign/ratio cases and separate a term's gradient, KL/entropy, and shared-parameter updates. Reference code was not executed.

PPO clipping: why not keep increasing the probability? · GSPO and ASPO: how should one answer share its gradient?

All five algorithm sections and the closing summary read, including PPO reward, return, and policy/value equations; not every figure audited. Site coverage separates roles from memory, per-prompt groups from batch statistics, clipping from hard constraints, and rule-based reward from hacking resistance. A DPO relative-probability counterexample extends the existing overview rather than duplicating it.

PPO, GRPO, and DPO: turning feedback into an update · DAPO: how sampling, length, and clipping change training · GSPO and ASPO: how should one answer share its gradient?

Prose and final reward formula read. New examples separate entropy from actual quality and test a sorting verifier with held-out evaluation. Performance percentages in reference figures were not individually audited or reused.

Alignment tax: what else changed when the model improved? · Verifiable rewards: did passing the tests complete the task?

Reference prose, the final stage-three reward formula, and length curves were read through the comments, completing the previously interrupted reading. Site notes distinguish manual modes, learned stopping, and external routing, retain primary-source AutoThink/RouteLLM checks and the stage-two reward correction, and do not claim training or benchmark reproduction.

Choosing an agent model: APIs, self-hosting, and routing · Qwen deep dive: one family across scales and reasoning modes · Looped Transformers: how many loops per token

Prose and 103-line example read; reference code not run and figures not exhaustively audited. Added entropy measurement, aggregation, binary derivatives, fixed-prefix controls, and paper-checked corrections to KL-Cov's old/reference distinction and absolute-difference approximation.

Alignment tax: what else changed when the model improved? · Three KL estimators: similar values, different gradients

All three sections and the final ratio example read. Ratios from one sequence do not prove variance or non-convergence. Original site examples already enumerate a two-step distribution and separate single-sample unbiasedness, variance, support and prefix correction; a local surrogate is not presented as full-trajectory IS.

On-policy and off-policy: can we train on old answers? · GSPO and ASPO: how should one answer share its gradient?

Reference prose read; embedded formulas not individually audited. Original Algorithm 1 grounds group sampling, fixed old denominators, Reference refresh, and asynchronous-reuse tradeoffs. Importance weighting does not automatically make old data on-policy.

On-policy and off-policy: can we train on old answers? · PPO, GRPO, and DPO: turning feedback into an update

Adds explicit eight-GPU dense TP8 / DP8 / DP2×TP4 weight accounting, request cooperation, and shared-EP exceptions. Teaching arithmetic only; no eight-GPU benchmark.

Distributed training: how are data, state, and computation divided?

All nine source stages and final diagnostics read. A new chapter covers the request path, chunk budgets, output, and cancellation against a pinned vLLM commit; no full service or performance benchmark was run.

What happens between an LLM request and the answer on screen?

Both sections and main equations read through the ending. Checked V3 §2.2 and DFlash §4: independent heads, shifted sequential-MTP inputs, and parallel drafting differ. Adds concatenation axes and a shape example; no reference-code execution or GPU reproduction.

DFlash: drafting, verification, and speed · Multi-token prediction: what do the extra targets teach?

Adds boundary bytes and lower bounds at an assumed 12 GiB/s, prefill/decode differences, serial request dependencies, and microbatch concurrency; PP is not assumed optimal.

Distributed training: how are data, state, and computation divided?

Prose and step-by-step flow read through the summary; embedded architecture artwork was not separately audited element by element. Site distinguishes draft/target KV and correction/bonus tokens, with sampling, rollback, and cost examples. Checked pinned official code and paper; no GPU performance reproduction.

DFlash: drafting, verification, and speed

A two-request reference table and pinned vLLM implementation distinguish cached/uncached queues, reference release, and mapping removal; no TTL or immediate GPU deallocation claim.

FlashAttention and PagedAttention: what does each save?

Pins vLLM and SGLang commits and compares a seven-token prefix at page sizes one and four; backend, hybrid-state, and unmeasured-throughput limits remain explicit.

FlashAttention and PagedAttention: what does each save?

A complete hypothetical budget, variable-length paging, physical sharing, and TP/PP/DP placement examples are tested. Source prose is read but its image equations remain unchecked; no GPU peak measurement.

KV cache: what do we stop computing, and start storing?

Reference table read. Adds two-rank concatenation, summation, destination, and ReduceScatter examples checked against NCCL semantics.

Tensor parallelism: two GPUs, one layer · Distributed training: how are data, state, and computation divided?

Source prose, tables, and conversion equation read. Adds weights, KV, workspace, and headroom; corrects weight-only deployment conclusions and GB/GiB ambiguity. Capacity figures are hypothetical.

KV cache: what do we stop computing, and start storing?

Reference prose and both figures read. New chapter traces attention/FFN forward and backward, separating the 2+2 baseline, inference, and SP rather than counting network packets.

Tensor parallelism: two GPUs, one layer · Distributed training: how are data, state, and computation divided?

Source prose read. Adds layout figures, last-valid-position code, and separate mask/position/loss responsibilities against a pinned Transformers commit; no actual generation model was run.

Decoder-only: autoregressive generation

Reference prose and both figures read. Adds a worked FFN, shapes, nonlinearity counterexample, input gradients, and CPU equivalence tests with a pinned Megatron revision.

Tensor parallelism: two GPUs, one layer · Distributed training: how are data, state, and computation divided?

Stage 1–3 prose and the memory-composition figure read in Chrome. A four-rank ring example gives 12/12/18 MiB sent, with precision, release, and gathering assumptions checked against ZeRO section 7. Equal bytes do not guarantee equal time; no multi-GPU benchmark.

Distributed training: how are data, state, and computation divided?

Reference prose and EOS diagram read. Original reports clarify training recipes, synthetic data, self-distillation, and embedding versus reranker objectives. No family-wide MLM-only claim or automatic quality advantage; no model training or GPU benchmark.

Choosing between Qwen3 Embedding and BGE-M3

Reference prose read through comments, with no code or figures remaining. Request flow, loading arithmetic, and permission tests use MCP 2026-07-28 and Skills specifications. No live server was run; protocol and instructions do not guarantee safety.

Agent tools: function calling, MCP, and skills · Agents: common building blocks

Read through code outputs and comments; equations inspected visually, reference code not executed. Added TF conventions, normalized duplication, and zero-IDF ties: output order is not stronger relevance, and b=0 cannot validate length normalization.

TF-IDF and BM25: how keyword retrieval scores a document

Reference prose and main equations read through comments. Candidate CE versus independent BCE now connects to target provenance, unit-vector examples, and alignment/uniformity assumptions; no binary-classification, human-label requirement, or guaranteed-uniformity claim.

Dual encoders: why encode queries and documents separately? · How CLIP matches images and text