Choosing an agent model: APIs, self-hosting, and routing

Extracting a date from a paragraph and debugging across several files do not necessarily need the same compute budget. One may need a short answer; the other may require reading code, running tests, and investigating the next failure.

Separate three decisions: which model to use, where it runs, and how much computation this request gets. Open-weight models can run behind hosted APIs. One model may support different reasoning budgets. An external router can choose between models. These are different controls.

For a practical starting point, read the comparison and the checklist. For automatic thinking, start with who decides. The four-request example is small enough to check by hand.

1. Start with whether the task gets done

Consider a notes-review assistant. One model may polish prose well but miss a sign error in an equation. Another calculates correctly but keeps rewriting passages the user wanted left alone. An aggregate benchmark score can hide both differences.

Collect a small, representative task set: definite mistakes, notes without errors, conflicting sources, and tool failures. Define completion before running candidates under the same conditions. Check the answer and whether the assistant read the right file, used the right version, and respected a review-only request.

Evaluate the whole execution for long tasks. Reading the wrong version early can send later reasoning in the wrong direction. Conversely, a failed step need not doom the task: the system might retry, use another tool, or ask for missing information.

2. What work comes with each deployment choice?

A small team might establish a baseline with an API that meets its data requirements rather than build serving first. A team with spare compute, deployment constraints, or steady traffic may start elsewhere. There is no universal request count at which self-hosting becomes the right choice.

3. Who decides whether to reason longer?

When a model offers a thinking mode, ask whether the caller selects it or the model chooses during generation. Both can be useful, but they are different capabilities. Why reasoning models get more answers right when they think longer, and when longer thinking stops helping, is covered in Reasoning models and test-time compute.

Caller-controlled modes. Original hybrid Qwen3 models support thinking/non-thinking controls. This lets the caller choose; it does not establish autonomous selection. The later Qwen3-4B-Instruct-2507 is non-thinking, so an earlier model's controls cannot be assumed for every checkpoint. Qwen3 announcement · Specific model card

Learned selection. AutoThink (2025) trains R1-style models to choose when to retain explicit reasoning. Its stages balance modes, improve performance, and introduce length-aware rewards. A prompt that produces both modes does not itself establish difficulty-aware allocation. AutoThink §2–3

A detail worth checking: stage 2 removes batch balancing but retains the mode-and-correctness-dependent base reward, rather than switching to binary accuracy reward. The main experiments concern mathematics. We have not reproduced them or established benefits for arbitrary agents.

When the first two layers merge. As of October 2026, some hosted APIs fold "the caller specifies" and "the model chooses" into one interface. In Anthropic's Claude API, for example, the caller sets an effort level (from low to max) and the model decides per request whether to think and how much. The documentation says the caller no longer sets a thinking token budget: max_tokens is the only hard cap, and effort is soft guidance that does not guarantee a token count. Claude docs: Steering thinking "Give date extraction a short budget" then becomes "give these requests a lower effort", and the model may still think longer on an occasional hard case. Check the thinking tokens actually reported per request type rather than trusting the configuration.

External routing. The application estimates which model suits a request; neither candidate needs a thinking switch. Adaptive depth instead concerns internal layers or iterations. Hidden-state computation is not the same as generating more text.

4. Routing and cascading decide at different times

FrugalGPT (2023) studies cascades; RouteLLM (2024, revised 2025) learns pre-generation routing from preferences. Their mechanisms are useful, but historical prices and savings do not establish today's deployment economics.

A wrong routing decision can be caught by output checks or fallback. Include those costs and delays in evaluation. Distillation can complement the serving strategies; it is not a fourth per-request allocation rule.

5. Four requests: spending on the wrong one buys nothing

Everything below is constructed, not a model benchmark. Each A call costs 1 unit, each B call costs 5, and routing adds 0.1 per request. B intentionally fails the fourth task to illustrate complementary strengths.

Both allocations call B once and cost the same. The mistaken one completes no more tasks than always using A. The challenge is not merely reducing expensive calls; it is spending them where they help.

The useful allocation was chosen after inspecting both outcomes. It is an idealized comparison, not information available at serving time. A real router must use information available before the call; outcome labels or future feedback would create an undeployable result.

import math

outcomes = [(1, 1), (1, 1), (0, 1), (1, 0)]
costs = {"A": 1.0, "B": 5.0}

def score_plan(choices, overhead=0.0):
    if len(choices) != len(outcomes) or any(name not in costs for name in choices):
        raise ValueError("Choose A or B for each request")
    if not math.isfinite(overhead) or overhead < 0:
        raise ValueError("Overhead must be finite and nonnegative")
    completed = sum(result[0 if name == "A" else 1]
                    for result, name in zip(outcomes, choices))
    cost = sum(costs[name] + overhead for name in choices)
    return completed, round(cost, 6)

assert score_plan("AAAA") == (3, 4.0)
assert score_plan("BBBB") == (3, 20.0)
assert score_plan("AABA", 0.1) == (4, 8.4)
assert score_plan("ABAA", 0.1) == (3, 8.4)

6. Count failed requests too

Log every input, output, tool call, and retry. Include money spent on failed tasks: averaging only successful requests hides expensive attempts that never finished.

For a simple estimate, assume fixed prices: $C_A$ per A call, $C_B$ per B call, $r$ for routing, and $v$ for checking an answer. Let $u$ be the fraction sent or escalated to B:

$$ \begin{aligned} C_{\text{route}}&=r+(1-u)C_A+uC_B,\\ C_{\text{cascade}}&=C_A+v+uC_B. \end{aligned} $$

The cascade has already paid for A. Its $u$ need not equal the router's. Real requests differ in length; accumulate actual input, output, and caching charges instead of assigning every call a fixed price.

7. Waiting, context, and versions also matter

Users wait for the whole task. Independent concurrent calls should not simply be summed; dependent calls sit on the critical path. Record queues, time to first token, generation, tools, and end-to-end median and tail latency. API clients can still adjust context, concurrency, timeouts, and output length.

Check what gets sent repeatedly. If every call resends the full history, starting with $b$ tokens and adding $a$ per step, then $k$ calls send:

$$kb+\frac{ak(k-1)}{2}.$$

For $b=1000,a=200,k=4$, the inputs contain 1000, 1200, 1400, and 1600 tokens: 5200 total. Summarization, windows, retrieval, and caching change storage, computation, and billing. Not every agent has quadratic cost. Outputs and tool returns need their own accounting.

Pin more than a model name. Weights, tokenizer, chat template, tool schemas, sampling, and serving versions can all affect results. After an upgrade, rerun representative tasks rather than only checking that the endpoint returns text.

8. What self-hosting estimates often miss

• Idle and peak capacity. Machines provisioned for peaks may keep billing through quiet periods. High average utilization can also mean requests are waiting.
• Prefill, decode, and KV cache. Long inputs, long outputs, and many concurrent requests stress different resources. Test the actual length distribution.
• Maintenance time. Someone must own upgrades, rollback, monitoring, on-call work, and recovery.

PagedAttention primarily improves KV memory management; RadixAttention focuses on reusable prefixes. They are not interchangeable and neither removes every limit. See one inference request and caching and attention kernels for mechanisms and sources.

For deeper systems implementation, also see Awesome-ML-SYS-Tutorial. This chapter establishes the questions; a cost figure alone is not a deployment design.

9. A practical order of checks

1. Define the task and boundaries. Which data is allowed, which actions need approval, and what failures are acceptable?
2. Establish a single-model baseline. Keep independent evaluation tasks before adding routing.
3. Find meaningful differences. Which requests benefit from more computation, and which simply lack information?
4. Compare complete systems. Include the router, checks, retries, and tail latency alongside quality.
5. Start small and reevaluate. Traffic, versions, and prices can change which allocation makes sense.

One final trap: low token entropy or a large top-1/top-2 probability margin indicates a concentrated next-token distribution, not a correct completed task. Before using either to trigger escalation, check its relationship to failures on held-out data. Confidently misunderstanding the request is not necessarily fixed by reasoning longer.