A car's front camera captures a frame. What is in it? Where is each thing in 3D? Is it the same thing as in the last frame? Those three questions go to detection, geometry and tracking. What these roles rest on is not any single network but two old crafts: geometry (converting between pixels and 3D) and estimation (combining repeated observations while tracking their uncertainty). In 2026 large models output cameras, depth and pointmaps directly, but reading, checking and using those outputs still takes these two.

Draft, pending review. The site owner has not reviewed this note yet; facts, emphasis and wording may change. Check the primary sources linked in the text before relying on it.

Why geometry and estimation cannot be skipped

  • A box lives only in pixels. The detector says "pedestrian at (860, 690)"; the planner needs "10 m ahead, 1 m to the left". Getting there takes the intrinsics, the extrinsics and one more constraint: the ground, a second camera or a depth model (note 1).
  • Small extrinsic errors get amplified. Several cameras can only be merged into a bird's-eye view once their extrinsics put them in one vehicle frame. Note 1 works it out: a 1° pitch error turns a ground-plane estimate of 10 m into about 8.9 m or 11.3 m.
  • Detection is per frame; planning needs tracks. Position, velocity, uncertainty and identity must persist across frames, which is filtering and data association (note 2).
  • New models output geometric quantities. Feed-forward models such as VGGT, Depth Anything 3 and MapAnything return camera intrinsics and extrinsics, depth and pointmaps in one forward pass (note 4); driving simulation with 3D Gaussian splatting first needs accurate camera models and poses (note 5).

Public job descriptions say the same. On 2026-10-15 we read 13 public job descriptions for cars, robots and humanoids and made a rough tally (whether an area is mentioned at all, not how much weight it gets): nearly every one asks for shipping to real vehicles, real robots or production code; about 8 mention real-time or hardware-aware optimisation; about 5 name 3D geometric vision, SLAM/VIO, sensor fusion or BEV. This series only aggregates knowledge areas; it names no company and covers no hiring process or interview content.

How the five notes divide the work

Note Question it answers Example worked through Prerequisites on this site
1 How a 3D point becomes a pixel What intrinsics, extrinsics and distortion each do; when one homography is enough; how triangulation recovers depth A point in the vehicle frame projects to pixel (860, 690); the ground and a second camera recover 10 m Linear algebra
2 State estimation and tracking How to keep position and identity across frames; why ByteTrack uses low-score boxes; filter-based and optimisation-based SLAM One Kalman predict-and-correct step in 1D; Hungarian matching on a 3×3 IoU matrix Probability (conditional probability, Gaussians); note 1
3 Detection and segmentation in 2026 How AP is computed; how DETR and YOLO26 drop NMS; choosing between closed-set real-time, open-vocabulary and SAM 3; which weights you can ship 3 ground-truth boxes and 5 predictions, all the way to AP@[.5:.95] ViT, CLIP
4 Frozen backbones and geometry foundation models Why self-supervised features do well on dense tasks; relative vs metric depth; what a pointmap is; how one forward pass returns cameras, depth and points The point (1, 0.5, 5) m scaled up 2× lands on the same pixel; fixing scale with one or two known distances; recovering the focal length from a pointmap ViT, CLIP; note 1
5 3D reconstruction and neural rendering What SfM and bundle adjustment solve; NeRF vs 3DGS; why 3DGS became driving-simulation infrastructure, and where it fails Two cameras compromising on one point's position; four Gaussians blended into one pixel; a pedestrian who only appears after a lane change Notes 1 and 4; the world-models series

Prerequisites on this site and outside textbooks

Reading paths by role

As of October 2026: what changed in the last two years

These are the main changes from 2024 to October 2026. All of them will age; cite them with their dates.

  • Segmentation takes a concept, not just a click. SAM 3 (arXiv 2025-11-20) takes a noun phrase or an exemplar and returns masks and IDs for every matching instance in images and video; SAM 3.1 (2026-03-27) adds multi-object tracking. The licence changed too: SAM 1 and 2 are Apache-2.0, while SAM 3 uses a custom SAM License (repository).
  • Self-supervised backbones reached 7B parameters, and dense features became the selling point. DINOv3 (2025-08-13) is a ViT-7B trained on LVD-1689M; Gram anchoring keeps dense features from degrading during long training. With a frozen backbone the authors report 66.1 mAP on COCO detection and 63.0 mIoU on ADE20k.
  • Depth went from one image to any number of views, with scale. Depth Anything 3 (2025-11-13) predicts consistent depth and rays from any number of views; the second version of MoGe adds metric scale, and MoGe-3 weights were released on 2026-08-18. Many of the strongest depth checkpoints have non-commercial licences.
  • Feed-forward 3D reconstruction became the main line. VGGT won the CVPR 2025 Best Paper; the CVPR 2026 Best Paper went to D4RT (dynamic 4D reconstruction), with an honourable mention for SAM 3D (CVPR 2026 closing page).
  • 3D Gaussian splatting became production infrastructure for driving simulation. NVIDIA Omniverse NuRec became generally available on 2026-03-25 (NVIDIA forum announcement); for Instant NuRec (2026-07-15), the authors report building a scene from a 10–20 s multi-camera log in about 1.5 s. Waymo takes the opposite view: it says its Genie-3-based generative World Model (2026-02-06) handles counterfactuals that leave the logged route, without a quantitative comparison (vendor-reported, Waymo blog).
  • Real-time detection split into two camps. The DETR-style RF-DETR (ICLR 2026) claims to be the first real-time detector above 60 AP on COCO, and DEIMv2 plugs in DINOv3 features; Ultralytics' YOLO26 (released 2026-01-14) dropped NMS, and YOLO27 is only a preview as of October 2026.
  • Open-vocabulary detection is merging into segmentation models and being tested on harder ground. The SAM 3 paper reports box AP on COCO and LVIS directly and calls these zero-shot results; RF100-VL shows vision-language detectors below 2% zero-shot on some medical datasets.
  • Benchmark culture changed. COCO box AP still appears in every detection paper, but as one column next to RF100-VL, ODinW13, LVIS or SA-Co; Papers with Code reportedly shut down in July 2025.
  • Check the licence before using weights. DUSt3R, MASt3R and CUT3R are CC BY-NC-SA 4.0; Fast3R and VGGT-Ω use the FAIR Noncommercial Research License; π^3 weights and CoTracker3 are CC BY-NC 4.0; the original 3DGS code has an Inria non-commercial licence; Ultralytics models are AGPL-3.0 or an Enterprise licence. The licence table for detection models is in note 3, and the one for depth and 3D models is in note 4.

Reading order

  1. How a 3D point becomes a pixel: the pinhole model, intrinsics and extrinsics, distortion, the epipolar constraint and triangulation, worked by hand on one point
  2. State estimation and tracking: the Kalman filter, Hungarian matching, ByteTrack's low-score boxes, and filter-based vs optimisation-based SLAM
  3. Detection and segmentation in 2026: AP by hand, NMS vs one-to-one matching, choosing between closed-set real-time, open-vocabulary and SAM 3, and licences of common weights
  4. Frozen backbones and geometry foundation models: DINOv3, Depth Anything 3, VGGT, and one forward pass that outputs cameras, depth and pointmaps
  5. 3D reconstruction and neural rendering: from SfM to Gaussian-splatting simulators, and where reconstruction is not enough