Perception & geometry: frozen backbones and geometry foundation models

You want to give a warehouse robot two skills: segment where on the floor it can drive, and estimate how far the obstacle ahead is. You have a few thousand labelled images, not enough to train a large backbone, so you plan to freeze a pretrained ViT and train only a small head on top. Should you pick a CLIP-style image–text model or a self-supervised model such as DINO? And can you read the numbers a depth model returns as metres? In the frozen-backbone comparisons the authors report, DINO-family features are clearly better than CLIP-style features on pixel-level tasks; relative depth from a single image is missing one scale factor by construction, which has to come from a known distance somewhere else; and since 2025, geometry foundation models return cameras, depth and points in one forward pass, but many of the strongest checkpoints cannot be used commercially.

Draft, pending review. The site owner has not reviewed this note yet; facts, emphasis and wording may change. Check the primary sources linked in the text before relying on it.

It helps to have read ViT, CLIP and this series' camera geometry note first: this note assumes you know the intrinsics $K$, the extrinsics $[R\mid t]$ and pinhole projection.

Freeze the backbone, train only a head

A frozen backbone means none of the pretrained weights change; the network is used only as a feature extractor. The image is cut into patches and the ViT outputs one vector per patch. Classification uses a single global vector (the CLS token or a pooled one). Dense tasks such as segmentation and depth need an answer for every pixel, so they use each patch's own vector: the patch features.

Backbones are often compared by linear probing: train only a linear layer on the frozen patch features. A high score means the needed information is in the features and can be read out linearly. The DINOv3 paper (Meta, 2025-08) compares backbones exactly this way in §6.1.2: segmentation is scored by mIoU on ADE20k (higher is better), depth by RMSE on NYUv2 (lower is better).

Why self-supervised features are stronger on pixel-level tasks

Start with the numbers. In Table 3 of the DINOv3 paper (frozen backbone, linear head, author-reported), on ADE20k the image–text contrastive SigLIP 2 g/16 scores 42.7 mIoU and PE Core G/14 scores 38.9; the self-supervised DINOv2 g/14 scores 49.5 and DINOv3 7B/16 scores 55.9. Depth RMSE on NYUv2 is 0.494, 0.590, 0.372 and 0.309 in the same order. The paper's own summary: on ADE20k, DINOv3 beats the self-supervised baselines by more than 6 mIoU and the weakly supervised (image–text) baselines by more than 13.

Why? Here is how the papers themselves explain it:

Two more overall discounts. First, all these numbers come from Meta papers, and PE, the model being compared against, is also Meta's; the evaluation uses a linear head and a fixed resolution (segmentation at 1024 patch tokens), so the table cannot tell you how much of the gap survives full fine-tuning. Second, if you need text queries ("find every pallet"), visual features alone are not enough: either use a CLIP-style model, or do what DINOv3's dino.txt does and train a text encoder to align with the frozen vision backbone (LiT-style training, DINOv3 §5.3).

Train longer and local features get worse

Self-supervision does not give good dense features for free either. DINOv3 §4.1 reports that while training a ViT-7B, ImageNet classification accuracy keeps rising, but linear segmentation on Pascal VOC starts falling after about 200k iterations; at the same time, the cosine-similarity map between one patch and all others gets noisier, and unrelated patches start to look alike. The image-level task and the pixel-level task are competing for the same representation.

The paper's fix is called Gram anchoring. Normalise each of an image's $P$ patch features and stack them into a $P\times d$ matrix $X$; $XX^\top$ then holds the cosine similarity of every pair of patches and is called the Gram matrix. During training, the student's Gram matrix is pulled towards that of an earlier checkpoint (the Gram teacher):

$$ \mathcal L_{\rm Gram}=\big\|X_SX_S^\top-X_GX_G^\top\big\|_F^2 $$

$X_S$ and $X_G$ are the student's and the Gram teacher's features, and $\|\cdot\|_F$ is the square root of the sum of squared entries. Multiplying all features by the same orthogonal matrix $Q$ leaves $(XQ)(XQ)^\top=XX^\top$ unchanged, so this loss only constrains "who is similar to whom", not where the features sit. The paper adds this term only after 1M iterations and refreshes the Gram teacher every 10k iterations.

One small calculation (a teaching example): three patches, a and b on the same pallet, c on the floor. Early features a=(1,0), b=(1,0), c=(0,1) give pairwise similarities ab=1, ac=0, bc=0. After long training they become a=(1,0), b=(0.8,0.6), c=(0.6,0.8), with ab=0.8, ac=0.6, bc=0.96: the floor now looks more like the pallet patch b than a and b look like each other. The diagonal is all 1, and each off-diagonal entry appears twice:

$$ \mathcal L_{\rm Gram}=2\big[(0.8-1)^2+(0.6-0)^2+(0.96-0)^2\big]=2\times1.3216\approx2.64 $$

The gradient pushes c away from b without asking any vector to return to its old coordinates.

Depth: relative depth is missing a scale

A pinhole camera maps a point $(X,Y,Z)$ in camera coordinates to pixel $(u,v)$:

$$ u=f\frac{X}{Z}+c_x,\qquad v=f\frac{Y}{Z}+c_y $$

$f$ is the focal length in pixels (here $f_x=f_y=f$) and $(c_x,c_y)$ is the principal point. Reuse the teaching camera from camera geometry: $f=1000$, $(c_x,c_y)=(960,540)$. The point $P=(1,\,0.5,\,5)$ m lands at $u=1000\times1/5+960=1160$ and $v=1000\times0.5/5+540=640$. Scale the whole scene about the camera centre by $k=2$: $P$ becomes $(2,1,10)$ and still lands on $(1160,640)$, because the $k$ in numerator and denominator cancels. One image corresponds to every scene whose depths are multiplied by $k$. No pixel in the image can tell them apart; a model can only guess metres from learned priors (how big cars usually are, how tall doors usually are).

That is why many models output only relative depth. Take three pixels in one image: the road, the car ahead and a distant building, with true depths 4, 8 and 20 m. A model that is off only by scale outputs $(1,2,5)$, the true values divided by 4. Now the lidar says the car is at 8 m: $k=8/2=4$, and the three pixels become $(4,8,20)$. A single known distance fixes the scale. It does not fix shape: if the model had said 4 instead of 5 for the building, $k=4$ from the car gives 16 m, and the 4 m error remains.

More common is affine-invariant output, where both the scale $s$ and the shift $t$ are unknown: $d=s\hat d+t$. Marigold and the first MoGe are affine-invariant. Suppose the model outputs $\hat d=(0,1,4)$. Knowing only that the car is at 8 m, the equation $8=s+t$ has infinitely many solutions: $(s,t)=(4,4)$ gives $(4,8,20)$ and $(2,6)$ gives $(6,8,14)$, and both agree with that one observation. Only when you also know the road is at 4 m do you get $t=4$, $s=4$ and a building at 20 m. Affine output needs two known distances. The Depth Anything 3 README points out that Depth Anything 2 outputs disparity (inverse depth): then $s$ and $t$ act on $1/d$, there are still two unknowns, but you cannot convert linearly in depth.

This figure needs JavaScript.

Where can a known distance come from?

• Sparse lidar: the Depth Anything V2 README (entry of 2024-12-22) mentions Prompt Depth Anything, which uses low-resolution lidar to "prompt" the model and get metric depth.
• A known camera height: assume the point is on a flat road, and the camera's height above it is the known distance. The camera geometry note works out 10 m with a camera 1.5 m above the ground, and how far that number moves if the pitch is off by 1°.
• A stereo baseline, several frames plus an IMU, or another metric model. The Nested version of Depth Anything 3 uses a multi-view model for pose and depth and a monocular metric model for scale (README).

A monocular model that outputs metric depth directly also has to deal with focal length. From $u-c_x=fX/Z$, doubling $f$ and every point's $Z$ leaves the pixels unchanged: a long lens filming a scene stretched along the optical axis produces the same image as a short lens filming the original. Without the focal length, metric depth is missing one condition. That is why Depth Pro (Apple, 2024-10) also estimates the focal length, Metric3D v2 first transforms images into a canonical camera space, UniDepth has a module that estimates its own camera, and MoGe-2 adds metric scale. The monocular metric version of Depth Anything 3 leaves the focal length to the user: its README gives "metric depth = focal length (pixels) × network output / 300", so a wrong focal length scales every distance by the same wrong factor.

Pointmap: a 3D point for every pixel

A depth map gives each pixel one number; a pointmap gives each pixel a 3D point. The DUSt3R paper (Naver Labs Europe, 2023-12) defines it in §3 as a $W\times H\times3$ array with a one-to-one mapping between pixels and scene points. Given depth $D$ and intrinsics $K$:

$$ X_{u,v}=D_{u,v}\,K^{-1}\begin{bmatrix}u\\ v\\ 1\end{bmatrix} $$

Continuing the example: pixel $(1160,640)$ at depth 5 m gives $K^{-1}[1160,640,1]^\top=((1160-960)/1000,\,(640-540)/1000,\,1)=(0.2,\,0.1,\,1)$; times 5 that is $(1,\,0.5,\,5)$, which is $P$ again.

In the other direction, the pointmap hides the camera. The point $(1,0.5,5)$ lands at $u=1160$; assuming the principal point is at the image centre, $f=(u-c_x)Z/X=200\times5/1=1000$. DUSt3R §3.3 estimates the focal length this way, with a robust fit over all pixels (assuming a centred principal point and square pixels). Two images are where it pays off: DUSt3R expresses both images' pointmaps in the first camera's coordinate frame, so where the same 3D point appears in both images is the correspondence; aligning the same image's points in the two frames (Procrustes in closed form, or the more robust PnP with RANSAC) gives the relative pose. Its output is only defined up to an unknown scale: the training loss first normalises prediction and ground truth by "the average distance of all points to the origin", which removes exactly the $k$ of the previous section.

Cameras, depth and points in one forward pass

The classical route matches features, runs structure-from-motion to get cameras, then does dense reconstruction, and every step is an iterative optimisation; the next note takes it apart. The line below turns it into one forward pass of a network:

A few things worth pausing on:

• The outputs are redundant. VGGT returns depth, cameras and a pointmap; its official README says that points unprojected from depth plus cameras are usually more accurate than those from the pointmap branch. The redundancy can be used as a consistency check.
• "Depth plus ray" is a factorisation. A 3D point equals the camera centre plus "distance along the ray times ray direction". Depth Anything 3 argues that predicting this one depth-ray target is enough; the MapAnything README outputs ray_directions, depth_along_ray, depth_z, camera_poses and pts3d separately.
• z-depth and distance along the ray are not the same thing. The example pixel's ray direction is $(0.2,0.1,1)$, of length $\sqrt{1.05}\approx1.025$. A z-depth of 5 m is a distance along the ray of $5\times1.025\approx5.12$ m. The further from the principal point, the bigger the gap. Unproject z-depth as if it were distance along the ray and the points near the image edges are pulled in wholesale (in the example the recovered z is only $5/1.025\approx4.88$ m); use distance along the ray as z-depth and they are pushed away (5.12 m).
• Check where the scale comes from, model by model. DUSt3R is up to scale; CUT3R and MapAnything output metric; Depth Anything 3 gets metric scale from its Metric and Nested versions.

All comparison numbers are author-reported. For example, the Depth Anything 3 abstract reports "surpassing prior SOTA VGGT by an average of 44.3% in camera pose accuracy …", and 25.1% on geometry, measured on a benchmark the authors built themselves.

Can you ship it? Licences of popular checkpoints

As of 2026-10-15, from each repository's licence file and the Hugging Face licence tags (this is not legal advice; licences change, so read the original text before shipping):

Licences really do change: VGGT added a commercial checkpoint on 2025-07-29 (README), and Segment Anything moved to a custom licence from SAM 3 on (see detection and segmentation). Within one model family, the small and large sizes often carry different licences.

Before you build on it

As of October 2026

• Successors: VGGT-Ω (2026-05) scales VGGT up and handles static and dynamic scenes; its arXiv comment says CVPR 2026 oral. As of 2026-10-15 we found no public release of a DINOv4 or a Depth Anything 4; not finding one does not prove there is none.
• Downstream use: DINOv3 is already used as an off-the-shelf backbone, for example by the DEIMv2 detector and the SAM 3D Body checkpoints; DINO-WM uses frozen DINOv2 patch features as a world-model representation (see the JEPA note).

Next, read 3D reconstruction and neural rendering: once you have cameras and points, how do they become a scene you can render from new viewpoints? Or go back to the series guide.