After deployment: requests, failures, and feedback

Original teaching design, checked 2026-10-11. Continues the document search example; times and traffic are hypothetical, not production measurements.

A model can look good offline and still disappoint users. Answers may be wrong, late, filtered down to too few results, or served from an old index. Make a request traceable before deciding to retrain.

1 · Admission: deciding whether to accept work

The client supplies a query, optional image, request ID, and related inputs. The server resolves identity and authorization, validates size, type, and length, and sets a deadline. Output-token limits and wall-clock deadlines are different controls.

Queues cannot grow indefinitely. Design rate limits, queue bounds, timeouts, and retries together. Three retries worsen overload if the original request is still running on the GPU. Pure reads may support deduplication or coalescing; tools with external side effects need their own idempotency and confirmation rules.

2 · Candidates: eligibility before ranking

Keyword, text-vector, and optional image–text retrieval can run in parallel. Each returns IDs, scores, provenance, and versions. Deduplicate, filter, fuse, and rerank under explicit rules. BM25, different encoders' cosine scores, and probabilities are not interchangeable numbers to sum blindly.

Where possible, authorization should constrain candidate eligibility. Post-filtering needs handling when fewer than k candidates remain. Recheck permissions and current revisions when fetching documents; an ACL indexed hours ago is not permanent authority.

3 · Generation: execution is not evidence validation

For a summary, the application selects authorized evidence, retains citation IDs, and builds the prompt. vLLM / SGLang handle generation scheduling and KV state. Instructions inside retrieved documents remain untrusted; entering a prompt does not grant tool permissions.

Streaming also requires a decision about what can be shown immediately and what needs checking first. Filtering a completed string cannot retract content already streamed to the user. The boundary depends on product risk; not every response needs full buffering.

4 · Latency: optimize what users wait for

For search without generation, imagine one serial request: admission 20 ms, query encoding 30 ms, retrieval 80 ms, reranking 60 ms, response 30 ms. That totals 220 ms, leaving 30 ms in a 250 ms budget. This is one illustrative path. Adding service p95s does not yield end-to-end p95, and parallel work must be analyzed on the critical path.

Now take a 500 ms request with 300 ms in model computation and 200 ms elsewhere. Making model computation three times faster gives $200+300/3=300$ ms overall: only $500/300\approx1.67$ times faster end to end.

def end_to_end_speedup(total_ms, accelerated_ms, factor):
    import math
    if not all(math.isfinite(value) for value in (total_ms, accelerated_ms, factor)):
        raise ValueError("expected finite values")
    if total_ms <= 0 or not 0 <= accelerated_ms <= total_ms or factor <= 0:
        raise ValueError("invalid timing or speed factor")
    new_ms = total_ms - accelerated_ms + accelerated_ms / factor
    return total_ms / new_ms


assert abs(end_to_end_speedup(500, 300, 3) - 5 / 3) < 1e-12

Load matters too. Near capacity, queueing can become substantial; multiplying a lightly loaded single-request time does not predict tail latency under concurrency. See the engine comparison protocol.

5 · Release a compatible bundle

Validate artifacts and run fixed-input regressions before shadow traffic or a canary. Shadow traffic helps compare outputs and execution, but shadow results did not receive real user reactions. Do not duplicate side-effecting tool calls. A canary changes some live traffic and needs predefined stop or rollback gates.

Models, processors, prompts, ranking configuration, and indexes need not all change together, but the release manifest must capture compatibility. Roll back a compatible combination, isolate cache versions, and confirm the previous version still has capacity.

A small project can start with old artifacts, a switchable pointer, fixed comparison inputs, and a rehearsed rollback. “Supports rollback” in a README is not a recovery test.

6 · Monitoring: not every drop is drift

A distribution change may reflect new users. A metric drop may be broken instrumentation. Check measurement and pipelines before choosing retraining. Delayed labels may require early proxies, but final conclusions need mature labels and stable definitions.

7 · Feedback: retain what was actually shown

A click should connect to the request, displayed list and positions, actual model / index revisions, exposure time, and experiment assignment. “User clicked document X” alone cannot reconstruct their alternatives or attribute a version change.

Where appropriate, sample actual serving features or revisions sufficient to reconstruct them, then compare training and serving. Google's Rules of ML discusses training-serving skew and feedback loops. Collection must still be minimized and respect authorization and retention limits.

Do not automatically treat previously displayed, clicked items as the only truth in the next dataset. The old model's omissions can become future blind spots. Use task-appropriate labeling, exploration, or audits; more logs alone do not remove selection bias.

8 · Diagnose across boundaries

Different teams can own different modules, but a service owner needs to coordinate diagnosis across the request. Useful end-to-end understanding means locating your layer's failures and communicating with upstream and downstream owners using inputs, versions, and traces.

For hands-on work, choose recommendation, RAG, or post-training. You do not need to build every path at once.