Last checked: 2026-09. Architecture explanations and invented examples, not an internal company implementation.
Start with a million possible items
A model can jointly examine a user and an item with rich interactions. But doing that for a million pairs on every refresh immediately raises questions about computation and latency.
The starting question for a dual encoder is simple: can work that doesn't depend on this request happen beforehand?
Encode the user as $q=f_\theta(x_u)$ and the item as $v_i=g_\phi(x_i)$, then connect them with a cheap score:
$$s(u,i)=q^\top v_i.$$
The important property is separability, not merely having two models. The towers are typically trained together but can encode independently at inference. They may share parameters or use different architectures. The official TensorFlow Recommenders tutorial demonstrates this factorization. Two-tower tutorial
What computation actually disappears?
flowchart TD
I["Item published or changed"] --> E["Item encoder"]
E --> V[("Versioned vector index")]
U["Request and user context"] --> Q["User encoder / compatible cache"]
Q --> S["Vector search"]
V --> S
S --> K["Smaller candidate set"]
K --> R["Richer ranking interactions"]
An item vector can serve many requests. Online work often still includes user encoding, feature reads, and search. Precomputed item vectors do not imply model-free online serving. User vectors can also be cached, with freshness consequences.
A rough comparison:
| Path | Main per-request work | What it saves and gives up |
|---|---|---|
| Joint encoding over the catalog | Run an interaction model for each user–item pair | Rich interactions, but cost grows with the candidate count |
| Dual encoder + exhaustive dot products | One user encoding and $N$ dot products of dimension $d$ | Reuses item encoding but still scans vectors |
| Dual encoder + ANN + ranking | User encoding, approximate search, then ranking a small set | Avoids some scanning but may miss exact neighbors |
An exhaustive dot-product scan costs $O(Nd)$ arithmetic, plus vector access. ANN cost depends on the algorithm, settings, and data; it cannot universally be called $O(\log N)$. Measure its quality–latency trade-off separately. Efficient retrieval and approximation
A storage calculation: vectors aren't free
An invented setting: 1 million items, 256 dimensions each, and 4 bytes per FP32 value.
$$M=Ndb=10^6\times256\times4=1{,}024{,}000{,}000\ \text{bytes}.$$
That's about 1.024 GB or 0.954 GiB for vectors alone. It excludes IDs, index structures, replicas, caches, and overlapping old/new deployments.
Lower precision, compression, and fewer dimensions can save space in different ways. Model parameter count is not the memory footprint of the retrieval pipeline.
Why not put every feature in the item tower?
Precomputing $v_i$ requires it to be independent of information that only arrives with a future request.
- Fits the item side: text, author or category features, and information that can be recomputed when the item changes.
- Fits the user side: profile, recent behavior, and available session context.
- Needs another treatment: whether this user just saw this item, their particular relationship with its author, and finer pairwise interactions. Some can be encoded in the towers; others belong in filters or ranking. Not every interaction fits losslessly into a dot product.
Making the item vector depend directly on each user sacrifices much of its cross-user reuse. That's possible, but it changes the cost model.
Items also change. How long an old vector may remain eligible after an edit is a product and engineering constraint, not something the encoder resolves.
Can one vector blend distinct interests?
Yes. A person might read both programming and photography. A single vector is convenient, but may not preserve useful neighborhoods for both.
Multi-vector or multi-source retrieval offers options. These two designs are different:
- Mix vectors, then search once: for example, $q=\alpha q_1+(1-\alpha)q_2$. Cost is easier to bound, but the interests may still be averaged together.
- Search separately, then merge: each vector retrieves its own neighborhood before deduplication and budget allocation. This retains distinct candidate sources at greater search and coordination cost.
The second is not a free upgrade. Fix the total candidate budget and measure unique relevant candidates, not simply how much more content extra searches return. Start with the complementarity example in episode 2.
Try it: mix first, or search separately?
The two interests point in different directions. Keep K=4 and weight=0.5, switch query modes, and compare the returned IDs. Then set K=8 and interest 1’s weight to 1 to see duplicates in split search.
This shows geometry and budgeting, not which candidates are more relevant. The two queries are also not the two towers: both are on the user side. The item tower still produces the catalog’s candidate vectors.
How do both towers learn a compatible notion of closeness?
Two towers separate the scoring computation, but don't yet specify which items should score well. One common training objective places a known positive above sampled alternatives:
$$\mathcal L_u=-\log\frac{\exp(s(u,i^+)/\tau)} {\exp(s(u,i^+)/\tau)+\sum_{j\in\mathcal N_u}\exp(s(u,j)/\tau)}.$$
Here $u$ is the user, $i^+$ a known positive, $s(u,i)$ the matching score, and $\tau>0$ the temperature. $\mathcal N_u$ is the sampled comparison set, not necessarily items the user explicitly rejected.
With one comparison item, equal scores assign probability 1/2 to the positive, giving loss $-\ln(1/2)\approx0.693$. If the positive's score exceeds the other's by $\tau\ln3$, its probability becomes 3/4 and loss is about 0.288. Training favors relative ordering within this comparison set; that probability is not the user's true click probability.
Other batch members' positives are convenient comparison items, but some may also interest this user: false negatives. Duplicate items, multiple positives, and the sampling distribution change the objective. In-batch negatives are drawn roughly in proportion to how often an item appears as a positive, so popular items are over-penalized; a common fix subtracts the log of each candidate's sampling probability from its logit (logQ correction). Multiple positives require adjusting the single-positive formula above.
A stronger tower may not fix missing label distinctions, coverage, or unsuitable negatives. Also distinguish contrastive retrieval training from generative next-token SFT: they don't have the same loss.
When should you not rush into two towers?
- The candidate set is already small: try direct scoring before taking on index maintenance.
- Exact terms matter: keyword or rule-based retrieval can be necessary complements.
- Fine-grained matching matters: retrieve first, then jointly encode a small set. This division also appears in text retrieval. Retrieve & Re-Rank
- Evidence for a complex model is weak: establish a measurable baseline and locate the bottleneck.
Two towers move much of the computation ahead of time but limit fine-grained user–item interaction. Whether the split is worth it depends on candidate scale, latency requirements, and how much the downstream ranker can recover.
How do multiple vectors stay affordable?
Multiple interests don't automatically justify many queries per request. Keeping profile and history vectors permits at least three distinct implementations:
| Design | Request-time work | What it preserves and gives up |
|---|---|---|
| Fuse into one weighted vector | One ANN query | Simple interface, but minority interests may still be averaged away |
| Query both, then merge | Two queries, deduplication, fixed total budget | Distinct neighborhoods at extra query and merge cost |
| Select a view or allocate quotas from current evidence | Routing followed by bounded queries | Budget control, but routing mistakes remove candidates |
Multi-tower training and multi-query serving are different choices. Two trained views may still produce one serving vector; simple separate-query experiments can test complementarity before complex training. Establishing that increment first is often more informative than building a full router immediately.
With a budget of 100, one query returning 100 versus two queries returning 100 each is not a matched baseline. Fix merged capacity and record ANN query counts and overfetch. Even with 100 final candidates, two queries may cost more. Caching user vectors saves encoding, not index searches or freshness maintenance.
Self-check
Both models output 256 dimensions. Can a new user tower use the old index?
Not necessarily. Equal dimensions don't imply a shared coordinate system. Check training compatibility, normalization, distance, and release versions.
Encode the user once, then score every item by dot product. Is that ANN?
No. That remains exhaustive exact search. Two towers describe model factorization; ANN describes the search implementation.