MoE: review questions

At fixed expert size, total expert parameters grow with N and main expert compute per token grows with k; the router still scores N experts. Mixtral 8x7B duplicates FFNs, not eight complete models, yielding 46.7B total and 12.9B active parameters rather than 8×7B.

A common router scores experts with a linear layer, then selects top-k. Mixtral renormalizes selected weights; Switch top-1 retains the selected full-softmax probability instead of turning its sole gate into one. Ordinary autograd does not differentiate discrete indices; task gradients flow through continuous gates. Unselected expert parameters receive no task gradient from that token, but their router logits may receive gradients through the full softmax or auxiliary objectives.

An expert that receives more tokens gets more training and can become even more likely to be chosen; this self-reinforcing imbalance can also leave devices waiting. Switch uses $\alpha N \sum f_i P_i$: assignment fraction $f_i$ is nondifferentiable, while mean router probability $P_i$ is differentiable. Gradients flow through $P_i$. The value is $\alpha$ at perfect balance, not a strict lower bound over all routing distributions.

Switch top-1 sets capacity from $\frac{\text{tokens}}{N}\times\text{CF}$, rounded according to the implementation. This caps each expert's capacity rather than guaranteeing equal runtime per device. An overflowing expert branch can be skipped while the token continues through the residual path. Top-k and dropless implementations may behave differently; one capacity formula is not universal.

The bias affects top-k selection while gate weights use original affinities. Overloaded experts have their bias reduced; underloaded experts have it increased. This update needs no auxiliary-loss gradient, but can change the selected set, outputs, and task gradients. V3 still retains a small sequence-wise balance loss ($\alpha=0.0001$), so it is not entirely free of auxiliary objectives.

Fine-grained experts allow more combinations of smaller experts at comparable expert matmul cost. More combinations do not guarantee better specialization or unchanged communication. Shared experts provide a common computation path; reducing duplication is a design aim, not a predetermined division of knowledge. DeepSeek-V3 uses both; Qwen3-235B-A22B / 30B-A3B have no shared experts.

Sparse activation does not automatically shrink weight storage by k/N. All parameters must live somewhere: GPU residency, device sharding, and offloading have different costs. Expert parallelism typically adds token dispatch and return traffic, while large batches may use many experts. Comparing memory or speed with a dense baseline requires matching quality, precision, batch, and deployment conditions.