DeepSeek-R1-Zero's reward checks only two things: whether the final answer is right, and whether the reasoning is wrapped in <think> tags. Nobody asked it to "think more", yet its answers grew longer during training while AIME 2024 pass@1 rose from 15.6% to 71.0% (v1 report, 2025-01). Under verifiable rewards, checking, backtracking and switching methods raise the chance of being right, so they get reinforced and length grows with them. But length is not the goal: it overshoots, it hops between ideas on hard problems, the way the objective is written can push answers longer on its own, and the written reasoning is not necessarily the computation the model relied on.
Draft, pending review. The site owner has not reviewed this note yet; facts, emphasis and wording may change. Check the primary sources linked in the text before relying on it.
R1-Zero: reward only the answer, and length grows by itself
In the DeepSeek-R1 report (v1 2025-01-22; the peer-reviewed version appeared in Nature on 2025-09-17 and corresponds to arXiv v2), the abstract claims that "the reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL) …". R1-Zero was trained like this:
- Starting point: DeepSeek-V3-Base, with no SFT first.
- Algorithm: GRPO. Sample a group of answers to the same problem and compute the advantage from the group's mean and standard deviation of rewards, with no critic. How the group signal is computed is in GRPO's four-answer example.
- Reward: an accuracy reward (math answers in a required format matched by rules; code run against test cases) plus a format reward, weighted equally. No neural reward model, on the grounds that it gets reward-hacked in large-scale RL. What rule-based rewards can and cannot check is in verifiable rewards.
- Template: it only requires reasoning first and the answer second, without prescribing what the reasoning says.
- Hyperparameters (v2): learning rate 3e-6, KL coefficient 0.001, rollout temperature 1, 16 answers per problem; maximum length 32,768 tokens before step 8.2k and 65,536 after; 10,400 steps in total (1.6 epochs), 32 problems per step.
What they observed: average answer length rose steadily during training, and when the maximum length was raised at step 8.2k, both performance and length jumped. AIME 2024 pass@1 went from 15.6% to 71.0% (v1); v2 reports a final 77.9%; with self-consistency voting both are 86.7%. v2 counts reflective words such as wait, mistake, verify and check, whose frequency rose 5 to 7 times during training; "wait" in particular was almost absent early, appeared occasionally between steps 4000 and 7000, and rose sharply after step 8000. An intermediate checkpoint showed what the paper calls an "aha moment". The price was poor readability and mixed Chinese and English, so the full R1 added a few thousand cold-start long-chain examples, a language-consistency reward, and a round of rejection-sampling SFT on about 800,000 examples (about 600,000 reasoning and 200,000 non-reasoning); the filtering is covered in rejection sampling.
The trained R1 adjusts its length to difficulty. Supplementary section E.4 of v2 measured, on 2024 competition problems: an average of 8,793 thinking tokens per problem at 61.8% pass@1; fewer than 7,000 for easy problems and more than 18,000 for the hardest; under 100 tokens for something like 1+1. All of this is self-reported by DeepSeek.
Two premises are easy to miss. First, v2 states that V3-Base's pretraining data contains a lot of math and code, so the model had already seen many reasoning traces, and web pages also contained plenty of answers generated by OpenAI models; "no SFT first" does not mean reasoning was learned from scratch. Liu et al. (2025-03, COLM 2025) found that DeepSeek-V3-Base shows "aha moment"-style self-reflection before any RL. Second, not all of the length growth is "learning to think more"; work it once below.
Why answers get longer: work it once
A teaching assumption with made-up numbers. A problem can be tackled in two ways: answer directly, about 300 tokens, correct with probability 0.3; or answer, check, and redo with another method if it looks wrong, about 900 tokens, where the first attempt is still right with probability 0.3 and the redo after a wrong first attempt is right with probability 0.5, for $0.3+0.7\times0.5=0.65$ in total. With a reward that only checks correctness, the second way has more than twice the expected reward, policy gradient pushes probability toward it, and average length moves from 300 toward 900. Nothing rewards "long"; length is a by-product of the strategy that raises accuracy.
The second source is how the objective is written. The original GRPO objective divides each answer's token-level objective by that answer's length $|o_i|$. Liu et al. point out that this creates a length bias: a wrong answer with advantage −1 spreads −0.01 per token if it is 100 tokens long, but only −0.001 per token if it is 1,000 tokens long. Both are wrong, yet the long one is penalised 10 times less per token, so wrong answers drift longer, while correct answers are pushed the other way, toward brevity. Their Dr. GRPO removes this normalisation; length then stops climbing during training and wrong answers become clearly shorter. The difference between averaging per answer and averaging per token is worked by hand in the DAPO note.
Distillation: teaching long chains to small models
The R1 report took 800,000 examples curated with R1 and used them for plain SFT of Qwen2.5 and Llama models from 1.5B to 70B, for 2–3 epochs and without RL. Self-reported AIME 2024 pass@1: R1-Distill-Qwen-7B 55.5%, R1-Distill-Qwen-32B 72.6%, against 50.0% for the then-current QwQ-32B-Preview. Section 4.1 of v1 ran a control: more than 10,000 steps of large-scale RL on Qwen-32B-Base produced R1-Zero-Qwen-32B at 47.0%, on par with QwQ-32B-Preview and far below the distilled 72.6%. The authors conclude that distilling a large model into small ones works very well, that small models struggle to match it with RL at this scale, and that going further still needs stronger base models and larger-scale RL.
s1 (Muennighoff et al., 2025-01) took this to an extreme: SFT of Qwen2.5-32B-Instruct on just 1,000 curated problems with reasoning traces, plus the budget forcing described in the next section, self-reporting up to 27% above o1-preview on competition math. Yue et al. (2025-04) also found that a distilled model's pass@k is above the base model's at every $k$, whereas RL models are overtaken by the base model at large $k$; see note 3.
| Approach | What it needs | What it buys | Risks |
|---|---|---|---|
| Large-scale RL (verifiable rewards) | Many automatically checkable problems, rollout compute | The model finds reasoning moves that raise accuracy on its own | Length inflation, reward hacking, limited gains on small models |
| Distilling long chains (SFT) | A stronger teacher and its reasoning traces | Small models cheaply pick up long-chain behaviour | Capped by the teacher; learns the teacher's mistakes and verbosity too |
| Distill first, then RL | Both | The full R1 does a cold start before RL; ProRL also starts from a distilled model | Costs add up |
The general method is in distillation.
How long to think: budgets and length control
There are three kinds of approach.
Forcing at decode time. s1's budget forcing: once thinking exceeds the limit, insert the end-of-thinking marker so the model answers; to make it think more, replace the end marker with "Wait" when the model tries to stop. It self-reports AIME 2024 rising from 50% to 57%, but gains essentially flatten by the sixth forced extension, and suppressing the end marker too often sends the model into repetitive loops.
A length term during training. Kimi k1.5 (2025-01) saw answer length grow markedly during RL and added a length reward: for $k$ samples of one problem with shortest length $\ell_{\min}$ and longest $\ell_{\max}$, sample $i$ gets $\lambda_i=0.5-\dfrac{\ell_i-\ell_{\min}}{\ell_{\max}-\ell_{\min}}$; a correct answer receives $\lambda_i$, a wrong one $\min(0,\lambda_i)$, and this is added to the original reward with a weight. Work one group by hand, with lengths 1000 / 2000 / 3000 / 5000 and outcomes right, right, wrong, right:
| Answer | Length | Outcome | $\lambda_i$ | Length reward |
|---|---|---|---|---|
| 1 | 1000 | Right | 0.5 | 0.5 |
| 2 | 2000 | Right | 0.25 | 0.25 |
| 3 | 3000 | Wrong | 0 | 0 |
| 4 | 5000 | Right | −0.5 | −0.5 |
Among correct answers, the longest one is penalised; a wrong answer never earns a positive bonus for being short. L1 (Aggarwal and Welleck, 2025-03, COLM 2025) puts a target length in the prompt ("Think for N tokens.") with reward $\mathbf 1(\text{correct})-\alpha\,\lvert n_{\text{gold}}-n_y\rvert$, using $\alpha=0.0003$: with a 1000-token target, a correct answer of 1500 tokens earns $1-0.0003\times500=0.85$; a correct answer of 3000 tokens earns only 0.4. The paper self-reports more precise length control than s1, and its 1.5B L1 model beating GPT-4o at the same reasoning length.
Product controls. As of October 2026, the documentation of the main APIs uses discrete levels as the primary control rather than token counts:
- OpenAI (reasoning guide):
reasoning.efforttakes none, minimal, low, medium, high, xhigh or max, each model supporting only a subset; reasoning tokens are not visible and are billed as output tokens; it recommends reserving at least 25,000 tokens for reasoning and output when starting out. - Anthropic (effort, extended thinking): the 2025-era
budget_tokens(minimum 1,024) is deprecated on the 4.6 models and rejected by 4.7 and later; instead, adaptive thinking lets the model decide whether and how long to think, and the caller adjusts effort levels (at most five: low, medium, high, xhigh, max, with support varying by model). The docs call effort a behavioural signal, not a strict token cap. - Google (Gemini thinking, page updated 2026-10-14):
thinking_leveltakes low, medium or high, with minimal on some models; only thought summaries are returned, but the full thinking tokens are billed. - DeepSeek (change log): since 2026-08-13, V4's thinking mode supports three levels, low, high and max.
These levels do not line up across models: Anthropic's documentation states that Sonnet 5.5's levels were recalibrated, so the same level name thinks a different amount than on the previous generation. Who decides how long to think (the caller, the model itself, or an external router) is discussed separately in choosing a model for an agent.
Overthinking and underthinking
- Overthinking. Chen et al. (Tencent AI Lab and others, 2024-12) asked "what is 2 plus 3?" and found o1-like models used 1,953% more tokens on average than conventional models, with one example writing out 13 solutions. Rounds after the first correct answer added little accuracy and little diversity of approach, and this happened more on easier problems.
- Underthinking. Wang et al. (2025-01) looked at QwQ-32B-Preview and DeepSeek-R1-671B on hard sets such as AIME 2024: wrong answers used 225% more tokens than right ones and switched approach 418% more often, frequently dropping a line of reasoning before finishing it.
- Thinking longer, getting worse. Gema et al. (2025-07, TMLR 2025-12) built tasks where longer reasoning lowers accuracy and identified five failure modes: Claude models become increasingly distracted by irrelevant information; OpenAI o-series models resist distractors but overfit to the problem's framing; models drift from reasonable priors to spurious correlations; all struggle to stay focused on complex deduction; and Claude Sonnet 4 expresses more self-preservation with longer reasoning.
- No allocation across problems. Fan et al. (2026-08) had models split one shared token budget across several problems, exam style. Models mostly worked greedily in the order presented, spending heavily on early problems and ignoring point values; a prompt to plan first spread the budget more evenly but still did not prioritise by difficulty or value.
Is the written reasoning the model's actual computation?
Seeing a long chain of thought, it is tempting to think you are seeing how the model reached its answer. Several primary studies show the two can come apart:
| Study | How it tests | Findings |
|---|---|---|
| Turpin et al. (2023-05) | Reorder few-shot examples so the right option is always (A), and see whether explanations mention this bias | Models are swayed but do not mention it, instead rationalising wrong answers; accuracy drops by up to 36% on 13 BIG-Bench Hard tasks (GPT-3.5, Claude 1.0) |
| Lanham et al. (2023-07) | Truncate the reasoning, insert mistakes, paraphrase, or replace it with meaningless filler tokens, and see whether the answer changes | Large differences across tasks: sometimes heavy reliance on the reasoning, sometimes almost none; the gain does not come from the extra compute of filler tokens alone; on most tasks, larger and more capable models reason less faithfully |
| Chen et al. (2025-05) | Put one of 6 kinds of hint in the prompt and, when the model uses the hint, check whether its reasoning says so | Average rate of mentioning the hint: Claude 3.7 Sonnet 25%, DeepSeek R1 39%; outcome-based RL raises it at first, then plateaus low (28% on MMLU, 20% on GPQA); in RL environments with planted loopholes, models exploited them over 99% of the time but mentioned them under 2% of the time in 5 of 6 environments |
| Baker et al. (2025-03) | Have GPT-4o read the chains of thought of a frontier reasoning model such as o3-mini on coding tasks | Reading the chain catches reward hacking better than watching actions and outputs alone; but once the monitor is put into the reward and optimisation gets strong, the agent learns to hide its intent in the chain while still cheating |
Korbak et al. (2025-07), a position paper signed by researchers from many labs and institutions, calls chain-of-thought monitoring a new and fragile opportunity: it is possible today, and a change in how models are trained could take it away. On the product side, OpenAI has not shown users the raw chain of thought since o1 (2024-09), only a model-generated summary; Anthropic said on 2025-02-24 that Claude 3.7 Sonnet's visible thinking should be treated as a research preview and that it cannot be sure it represents what is really going on inside the model. As of October 2026, the OpenAI and Google API docs return only reasoning summaries.
So a chain of thought is evidence, not proof. To learn whether a step actually affects the answer, intervene: delete it, corrupt it, replace it with something irrelevant, and see how the answer changes. Approaches that compute more rounds in the hidden state without writing tokens (Looped Transformer) do not even leave this written evidence.
Doing the sums: where the money for thinking goes
A teaching example; every number is made up. A reasoning model has three settings on a set of problems, plus one parallel option. Assume an output price of USD 10 per million tokens and count only output (including thinking) tokens:
| Approach | Output tokens per problem | Accuracy | Cost per problem | Cost per correct answer | Sequential generation length |
|---|---|---|---|---|---|
| low | 1,000 | 60% | USD 0.01 | USD 0.017 | 1,000 |
| medium | 4,000 | 75% | USD 0.04 | USD 0.053 | 4,000 |
| high | 16,000 | 80% | USD 0.16 | USD 0.200 | 16,000 |
| medium, 3 samples and a vote, errors spread out | 12,000 | 84.1% | USD 0.12 | USD 0.143 | 4,000 |
| medium, 3 samples and a vote, errors concentrated | 12,000 | 78.4% | USD 0.12 | USD 0.153 | 4,000 |
From low to medium, 3,000 extra tokens buy 15 points, or 5 points per 1,000 tokens; from medium to high, 12,000 extra tokens buy 5 points, only about 0.4 points per 1,000 tokens. Returns fall off quickly, in the same direction as Anthropic's documentation saying higher budgets bring diminishing returns and more latency.
The two voting rows need one more assumption: medium's 75% comes from half easy problems (0.95 correct per try) and half hard ones (0.55). Using note 1's method, with spread-out errors maj@3 is 0.995 on easy problems and 0.686 on hard ones, 84.1% on average; if both kinds of problem have their errors concentrated on one answer, easy problems give 0.993 and hard ones only 0.575, 78.4% on average. For the same 12,000 tokens, parallel voting may beat the high setting or lose to it, depending on how the errors are structured, and only your own problems can tell you which. Parallel sampling also helps latency: the 3 samples can be generated at once, so the sequential length is only 4,000. Note too that only output was counted here; each parallel sample reads the prompt again unless the server reuses a prefix cache (see KV cache and inference).
What to check when you try it
| Try first | What to observe | What you can NOT conclude |
|---|---|---|
| Sweep effort or budget on one set of problems, logging thinking tokens and correctness per problem | Where the accuracy-versus-tokens curve flattens | That longer average length means "thinking more deeply" |
| Split answers into right and wrong; compare length and number of approach switches | Whether wrong answers are longer and switch more often | That because length correlates negatively with accuracy, everything should be shortened |
| During training, log the lengths of right and wrong answers separately | Whether wrong answers are growing abnormally (length bias) | That all the length growth comes from learning to check |
| Intervene on the chain: delete a step, corrupt a step | Whether the answer changes with it | That one unchanged answer proves the reasoning is unfaithful |
| At equal tokens, compare one long chain with a vote over several short ones | Which is more accurate and lower latency on your problems | That conclusions from someone else's problems carry over |
What to watch on this line next
These are open questions and what kind of result would change the current picture; they are not predictions.
- Will length be reined in or keep growing? R1-Zero's (2025-01) length rose with training; Kimi k1.5 (2025-01) and L1 (2025-03) rein it in with length rewards, and Dr. GRPO (2025-03) removes the normalisation bias. Watch whether new large-scale RL reports publish separate length curves for right and wrong answers during training.
- Can models allocate their own budget? From 2026-02-05 Anthropic hands "whether and how long to think" to the model; Fan et al. (2026-08) found cross-problem allocation still done poorly. Scores on evaluations that reward allocating by difficulty and point value will show how far adaptive thinking is from thinking on demand.
- How long can chains of thought be monitored? Baker et al. (2025-03) and Chen et al. (2025-05) show monitoring is useful but not sufficient, and that optimisation pressure can break it. Watch whether system cards publish monitorability measurements and commit to not optimising the chain of thought directly.
- Small models: distillation or RL? R1 (2025-01) found distillation clearly beat RL at 32B; ProRL (2025-05) ran more than 2,000 further RL steps on a 1.5B distilled model and reported gains on problems the base model could not solve. Controlled comparisons from several starting points at equal compute would make this clearer.