Scaling: review questions

For each training token a dense Transformer does about $2N$ FLOPs in the forward pass (one multiply-add per weight) and about twice that in the backward pass, $6N$ in total, multiplied by the number of tokens $D$. It drops the attention term that grows with context; Kaplan's $N$ excludes embeddings, Chinchilla's includes them. The budget is GPUs × peak × MFU × time: $64\times989.4$ TFLOPS (BF16 dense) $\times0.4\times3.95\times10^{5}$ seconds $\approx10^{22}$ FLOPs. At 20 tokens per parameter, $N=\sqrt{C/120}\approx9.1\times10^{9}$ and $D\approx1.8\times10^{11}$. Using the sparse peak would overstate the budget by a factor of two.

Kaplan (2020) found $N_{\rm opt}\propto C^{0.73}$; Chinchilla (2022) found 0.46–0.50. Chinchilla points out that all of Kaplan's runs used one fixed-length cosine, so when losses were read off midway the learning rate had not yet decayed, the losses were too high, and the effect of less data was underestimated; Kaplan's models were also small. Pearce and Song show that counting only non-embedding parameters and fitting at small scale pushes the exponent up. Porian et al. start from 0.864 on OpenWebText2 and, correcting last-layer compute, warmup, cosine decay and per-scale hyperparameters (including AdamW's $\beta_2$) in turn, get down to 0.518, and find that Chinchilla's result holds even without learning-rate decay.

It comes from Table 3 of Chinchilla's approach 1 (for example, 1B parameters with 20.2B tokens). Conditions: MassiveText, data seen about once; $N$ includes embeddings and tokens use their tokenizer; a cosine decayed 10x and matched to the run length. The original approach-3 fit favours smaller models, but Besiroglu et al.'s refit also returns to about 20; DeepSeek LLM found that data quality moves the exponent from 0.450 to 0.578. The optimum is flat: on the refit surface at $10^{22}$ FLOPs, a model half or twice the optimal size has a loss only about 0.01 higher, and 15%–26% more compute makes up the gap.

IsoFLOP: 3–5 small budgets, 5–10 sizes each, schedule matched to run length; fit a parabola to loss versus $\log N$, take its minimum, then fit a line through the minima on log–log axes. The teaching points from $(10^{18},9\times10^7)$ to $(10^{21},2.9\times10^9)$ give a slope of about 0.50 and extrapolate to about 9.2B parameters at $10^{22}$. An exponent error of ±0.04 changes a 10x extrapolation by only about 10%, but a 1000x extrapolation ranges from 70B to 121B. Risks: the frontier bends (Chinchilla Appendix E), the law must fail somewhere (Kaplan §6.3), seed noise is about 0.02, and smooth loss does not mean smooth downstream metrics. The successful predictions by GPT-4, DeepSeek LLM and Llama 3 are all self-reported.

Chinchilla minimizes training compute only. Sardana et al. fix a target loss and minimize $6ND_{\rm tr}+2ND_{\rm inf}$: the more inference, the smaller the model and the more data. On the refit surface, targeting the loss of the compute-optimal model at $10^{22}$ FLOPs, $D_{\rm inf}=10^{12}$ gives an optimum of about 4.6B parameters and 447B tokens: 24% more training compute, 23% less in total. Sardana et al. also found that fits on typical ratios overestimate the value of data at extreme ratios, so the savings at very large inference volumes are optimistic upper bounds. The Llama 3 report self-reports that the smaller models were trained far past compute-optimal and do better at the same inference budget, while 405B sits close to its own fitted optimum (402B, 16.55T).

Muennighoff et al. convert repetition into effective data: $D'=U_D+U_DR_D^*(1-e^{-R_D/R_D^*})$ with $R_D^*\approx15.4$. 100B unique tokens trained for 4 epochs is worth about 372.6B (93%); 16 epochs about 1.06T (66%); the ceiling is about 16.4 times. The paper concludes that at fixed compute, repeating about 4 times is nearly as good as new data (an 8.7B model at 4 epochs had only about 0.5% higher validation loss), after which returns fall fast. The decay constant for excess parameters, $R_N^*\approx5.3$, is smaller, so when data is limited more passes beat a bigger model.

McCandlish et al.: $S=S_{\min}(1+B_{\rm crit}/B)$, $E=E_{\min}(1+B/B_{\rm crit})$, $B_{\rm crit}=E_{\min}/S_{\min}$, roughly the gradient noise scale. With the teaching value $B_{\rm crit}=2$M, going from 4M to 16M saves only 25% of the steps but costs 3 times the data. It grows as loss falls (Kaplan: it doubles for about every 13% drop in loss), and recent work finds it grows mainly with data size (Zhang et al.: with data fixed, growing only the model gives an exponent of just 0.087). Conversion: the linear rule for SGD; for Adam the square-root rule (learning rate times $\sqrt\kappa$, $1-\beta$ times $\kappa$), valid only in the noise-dominated small-batch regime. DeepSeek LLM's fit can serve as a starting point: about $5.5\times10^{-4}$ and 4.6M tokens at $10^{22}$ FLOPs.

Under the standard parametrization (SP) the optimal learning rate moves with width (about an order of magnitude from width 256 to 8192 for TP5's MLP). Adam's first update is about $-\eta\,{\rm sign}(g)$ and is aligned with the input $x$, so $(\Delta Wx)_i\approx\eta\,n\,\mathbb E|x|$: with $x=\pm1$ and $\eta=0.01$, the change is 5.12 at width 512 and 40.96 at width 4096. µP makes the Adam learning rate of hidden and output layers $\propto1/\text{fan\_in}$, sets the output initialization variance to $1/\text{fan\_in}^2$, and uses $1/d$ attention, so the hidden learning rate at width 4096 is 0.00125. What does not transfer: regularization such as dropout and weight decay; depth, batch size and training length are only validated empirically, and multi-layer Transformer blocks face fundamental limits in depth (TP6). Everett et al. found that SP with the right per-layer learning rates also transfers.