How CLIP matches images and text

Take three images—a cat, a bicycle, and a coffee—and their three captions. Instead of asking for an essay, ask: which image belongs with which caption? This is a useful starting point for CLIP.

First review vectors, cosine, and softmax. This note covers the original 2021 CLIP objective, not a universal recipe for multimodal models.

Encode separately, then compare

We first establish the softmax matching objective. The next note, SigLIP, uses a small score table to compare pairwise sigmoid loss and introduces SigLIP 2 extensions.

CLIP processes images and text with separate encoders, projects to a shared dimension, normalizes, and scores pairs. The paper studied both ResNet and ViT vision branches; CLIP is not synonymous with one vision architecture.

flowchart TD
    A["Image batch"] --> B["Image encoder"]
    C["Caption batch"] --> D["Text encoder"]
    B --> E["Project + normalize: N × d"]
    D --> F["Project + normalize: N × d"]
    E --> G["Pair scores: N × N"]
    F --> G
    G --> H["Rows find captions; columns find images"]
    H --> I["Symmetric loss updates both encoders"]

Image tokens do not cross-attend to every caption here. Independent encoding helps retrieval, at the cost of compressing interaction details into vectors.

Start with a score table

These are invented teaching scores, not model measurements. Rows are images; columns are captions. Dataset pairs lie on the diagonal.

To retrieve a caption for the cat image, compare row 1. To retrieve an image for the cat caption, compare column 1. Column probabilities need their own normalization; transposing row probabilities is not equivalent.

Write the operation as a loss

For normalized image and text vectors $u_i,v_j$, let $s_{ij}=u_i^\top v_j/\tau$. The row objective is:

$$L_{I\to T}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(s_{ii})}{\sum_{j=1}^{N}\exp(s_{ij})}.$$

The column objective swaps queries and candidates; $L=(L_{I\to T}+L_{T\to I})/2$. This is the paper's symmetric cross-entropy objective. The official implementation exponentiates a learned logit scale, corresponding to $1/\tau$ here.

For row 1 at $\tau=1$, the paired probability is about 0.489 and its loss about 0.716. Set the lab above to τ = 1.00 with the cat image selected to see the same 48.9% and 0.716; the script below prints the row-1 probabilities and the batch symmetric loss:

python 00-foundations/code/multimodal_math.py
python 00-foundations/code/multimodal_math.py --torch

The first needs no PyTorch. The second checks the same scores with PyTorch, then updates toy vectors. It demonstrates loss and gradients, not training a real CLIP model.

Three easy mistakes

First clarify “unlabeled”: CLIP does not require a human class ID for each image, but image–text pairing is supervision. Noisy web descriptions are not arbitrary text without a learning target. Natural-language supervision relaxes fixed categories while making pair quality part of the training problem. CLIP paper

1. “Every other pairing is wrong.” That is the single-positive target assignment, not a real-world guarantee. The duplicate-caption setting uses two cat images, each matching both identical cat captions, while the target accepts only one. Equal scores split probability, so each approaches at most 0.5 and the image-to-text pair loss cannot reach 0. The other valid caption becomes a false negative. This is separate from the wrong-score scenario where a cat image favors a bicycle caption.
2. “Gradient accumulation enlarges the contrastive pool.” If each microbatch scores only its own candidates, cross-microbatch pairs never enter the denominator. Accumulation does not add them automatically; cross-device candidate sharing also requires an explicit implementation.
3. “Lower temperature is always better.” It makes the highest score more decisive, including when that score is wrong. Temperature cannot repair incorrect pair labels.

Where zero-shot classification comes from

Encode candidate descriptions such as “a photo of a cat,” then compare them with the image. Zero-shot here means no weight update for the downstream classification task—not no pretraining or a guarantee that related concepts were unseen. See the official CLIP example.

Original CLIP scores classes or pairs; it does not generate an answer token by token. To ask what the person on the left is doing, continue with connecting vision to a language model.

What does a score of 0.88 mean?

Suppose a bicycle photo has logits [2, 0] for two candidates. Softmax assigns about 0.881 to the first. Add another candidate scoring 2, giving [2, 0, 2], and the first probability falls to about 0.468. The image, original candidates, and original scores have not changed.

This invented softmax example shows candidate-list dependence, not an “88% chance of identifying the real object correctly.” Rejection and confidence thresholds require calibration under a fixed protocol, especially when none of the candidates fits.

CLIP's text branch produces text vectors, but cross-modal training does not establish suitability for every text–text task. For sentence retrieval, compare against a dedicated text-embedding baseline on the target data. Being able to compute cosine does not establish that cosine captures the semantics you need.