Perception & geometry: how a 3D point becomes a pixel, and back

In the front camera, a pedestrian's foot lands on pixel (860, 690). How far away is the pedestrian? That one pixel cannot tell you: projection throws depth away, and a pixel corresponds only to a ray. To get depth back you must add one more constraint: the point lies on a known plane, a second view sees it, or a learned depth prior fills it in. This note follows a single point through forward projection, back-projection, distortion, two views and triangulation. Linear algebra is all you need.

Draft, pending review. The site owner has not reviewed this note yet; facts, emphasis and wording may change. Check the primary sources linked in the text before relying on it.

Which frames does a point pass through?

A point passes through three coordinate frames:

• World frame (here, the vehicle frame): x forward, y left, z up, in metres.
• Camera frame: origin at the optical centre, x right, y down, z forward along the optical axis. This is the usual convention in vision; with another convention the signs below change.
• Pixel frame: u to the right, v down, origin at the top-left corner of the image.

The extrinsics $[R\mid t]$ take world coordinates to camera coordinates, and the intrinsics $K$ project camera coordinates onto pixels:

$$ \lambda\begin{bmatrix}u\\v\\1\end{bmatrix}=K\,[R\mid t]\begin{bmatrix}X\\Y\\Z\\1\end{bmatrix},\qquad K=\begin{bmatrix}f_x&s&c_x\\0&f_y&c_y\\0&0&1\end{bmatrix}. $$

$f_x,f_y$ are focal lengths in pixels, $(c_x,c_y)$ is the principal point, and $s$ is the skew term for pixel axes that are not perpendicular, usually set to 0. $\lambda$ is the point's depth $Z_c$ in the camera frame: the two sides agree only up to a scale, and that is exactly where depth is lost. Zhang 2000, Section 2, uses this pinhole model (he writes $K$ as $A$; the technical-report version is free to read). The Hartley & Zisserman sample chapter 9 writes the camera matrix as $P=K[R\mid t]$.

$R$ rotates the world axes onto the camera axes; $t=-RC$, where $C$ is the optical centre's position in the world frame. Reading $t$ directly as "where the camera is mounted" is an easy mistake.

Work it once by hand: from vehicle frame to pixel

Mount the camera at $C=(2,0,1.5)$ in the vehicle frame: 2 m ahead of the rear axle, 1.5 m above the ground, optical axis pointing straight ahead. The image is 1920×1080, $f_x=f_y=1000$, $(c_x,c_y)=(960,540)$, $s=0$. These are teaching values, not any real car.

The vehicle's "forward, left, up" must become the camera's "right, down, forward": camera x = −vehicle y, camera y = −vehicle z, camera z = vehicle x. So

$$ R=\begin{bmatrix}0&-1&0\\0&0&-1\\1&0&0\end{bmatrix},\qquad t=-RC=\begin{bmatrix}0\\1.5\\-2\end{bmatrix}. $$

The pedestrian's foot is at $X_w=(12,1,0)$ in the vehicle frame: 12 m ahead, 1 m to the left, on the ground.

1. Extrinsics: $X_c=RX_w+t=(-1,0,12)+(0,1.5,-2)=(-1,\,1.5,\,10)$, i.e. 1 m left of the camera, 1.5 m below it, 10 m ahead. That matches intuition.
2. Divide by depth: $(X_c/Z_c,\;Y_c/Z_c)=(-0.1,\;0.15)$. These are normalised coordinates.
3. Intrinsics: $u=1000\times(-0.1)+960=860$, $v=1000\times0.15+540=690$.

The pixel is (860, 690). Now change one condition at a time:

Backwards: a pixel gives only a ray

Back-projection first removes the intrinsics with $K^{-1}$:

$$ \begin{bmatrix}x_n\\y_n\\1\end{bmatrix}=K^{-1}\begin{bmatrix}u\\v\\1\end{bmatrix} =\begin{bmatrix}(860-960)/1000\\(690-540)/1000\\1\end{bmatrix} =\begin{bmatrix}-0.1\\0.15\\1\end{bmatrix},\qquad X_c=Z_c\begin{bmatrix}x_n\\y_n\\1\end{bmatrix}. $$

Any positive $Z_c$ projects to the same pixel. There are three ways to pin it down:

• A known plane. Assume the point is on the ground. The camera is 1.5 m up and its y axis points down, so the ground is $Y_c=1.5$; then $0.15\,Z_c=1.5$ and $Z_c=10$ m, which is correct. But the point 0.5 m above the ground sits at v=640, and the same assumption gives $Z_c=1.5/0.1=15$ m, 50% too far. The plane assumption holds only for points that really are on the plane.
• A second view. The point is where the two rays meet; see triangulation below.
• A learned prior. A monocular depth network learns from many images "how far things usually are". Relative depth gives only order and ratios; metric depth gives metres. Those models are in note 4.

The 1° pitch change from the table turns 10 m into about 8.9 m or 11.3 m under the ground assumption (depending on the direction). A system that measures distance from the ground plane must either estimate camera pitch online or use a cue that does not depend on the ground.

Lens distortion: worse towards the edges

Real lenses are not ideal pinholes. Zhang 2000, Section 3.3, keeps only the first two radial-distortion terms, applied to normalised coordinates:

$$ \breve x=x\,(1+k_1r^2+k_2r^4),\qquad \breve y=y\,(1+k_1r^2+k_2r^4),\qquad r^2=x^2+y^2. $$

Take $k_1=-0.2$, $k_2=0$ (barrel distortion, a teaching value). Our point has $r^2=0.0325$, a factor of 0.9935, so the pixel moves by only about 0.65 px and 1 px. A point near the right edge, $(0.9,\,0.5)$, has $r^2=1.06$ and a factor of 0.788: u moves from 1860 to about 1669, roughly 190 px.

So undistort first, then apply the pinhole equations. Wide-angle and fisheye lenses need their own camera models; ORB-SLAM3, for example, supports both pinhole and fisheye models. The original 3D Gaussian splatting rasteriser assumes an ideal pinhole; NVIDIA's 3DGUT adds distorted and rolling-shutter cameras (the paper's claims here are qualitative).

Calibration means estimating $K$, the distortion coefficients and the extrinsics. Zhang's method only needs the camera to see a planar pattern from at least two different orientations; it solves in closed form, then refines by maximum-likelihood nonlinear optimisation. Multiple sensors also need their time offsets calibrated: at 20 m/s, a 10 ms offset is a 0.2 m position error. Furgale et al. 2013 calibrate time and space together.

Two views: the epipolar constraint turns a 2D search into a line

Add a second camera with the same $K$, shifted 0.5 m to the right and not rotated: a stereo pair. In the first camera's frame, the second camera has $R=I$ and $t=(-0.5,0,0)$. The same point is $(-1.5,\,1.5,\,10)$ in its frame, at pixel (810, 690).

Without knowing depth, where can the first image's point land in the second image? Its ray projects to a line in the second image, the epipolar line. With the essential matrix:

$$ E=[t]_\times R,\qquad [t]_\times=\begin{bmatrix}0&-t_z&t_y\\t_z&0&-t_x\\-t_y&t_x&0\end{bmatrix},\qquad \hat x'^{\top}E\,\hat x=0, $$

where $\hat x,\hat x'$ are normalised coordinates in the two images (HZ chapter 9, eq. 9.11). With our numbers, $[t]_\times$ has only 0.5 in row 2, column 3 and −0.5 in row 3, column 2, so $E\hat x=(0,\;0.5,\;-0.075)$. Its dot product with $\hat x'=(-0.15,\,0.15,\,1)$ is $0.15\times0.5-0.075=0$: the constraint holds.

$E\hat x$ is itself the epipolar line in the second image: $0.5\,y'-0.075=0$, i.e. $y'=0.15$, pixel row $v'=690$. You only need to search along row 690, so a 2D search becomes 1D.

This is the simplest case. With rotation, epipolar lines are tilted. Under pure forward translation the epipole $e'=K't$ sits exactly at the principal point (HZ eq. 9.3), and all epipolar lines radiate from the image centre; a monocular camera on a car driving straight is close to this. Without $K$, write the fundamental matrix in pixel coordinates, $F=K'^{-\top}EK^{-1}$, and the constraint becomes $x'^{\top}Fx=0$. HZ chapter 9 gives $F$ 7 degrees of freedom and $E$ 5, and notes that Longuet-Higgins 1981 introduced the essential matrix to computer vision (a similar matrix appeared earlier in the photogrammetry literature); $F$ generalises it by dropping the assumption that the cameras are calibrated.

In practice $E$ or $F$ is estimated from matched points. Each match gives one linear equation in the 9 entries of $F$; with 8 or more matches you can solve linearly (the eight-point algorithm) and then enforce rank 2. Hartley 1997 states in its abstract that a simple normalisation of the matched coordinates (translation and scaling) beforehand makes the eight-point results comparable to the best iterative algorithms of the time. Wrong matches are removed with robust methods such as RANSAC (Barfoot, 2nd ed., Section 5.4.1).

Triangulation: getting depth back

The 3D point is where the two rays meet. A stereo pair with only a horizontal shift is the easiest case: HZ eq. 9.6 gives $x'=x+Kt/Z$ for pure translation, and substituting $t=(-b,0,0)$ gives

$$ d=u-u'=\frac{f\,b}{Z}\quad\Longrightarrow\quad Z=\frac{f\,b}{d}. $$

Our point has disparity $d=860-810=50$ px, so $Z=1000\times0.5/50=10$ m, matching where we started.

Now the error. Differentiating $Z=fb/d$ with respect to $d$ gives $|\partial Z/\partial d|=Z^2/(fb)$. At 10 m, a 1 px disparity error is a 0.2 m depth error; at 40 m the disparity is only 12.5 px and a 1 px error becomes 3.2 m. Triangulated depth error grows with the square of distance, which is the direct reason stereo and multi-view ranging get worse far away. A longer baseline $b$ or focal length $f$ helps, at a price: the two cameras share less field of view up close, and the two images look more different, so matching gets harder.

This figure needs JavaScript.

The general case (rotation, more than two views) has no formula this tidy. With noise the two rays do not meet. The usual approach is a linear method for a first estimate, then minimising reprojection error: the 3D point should project back as close as possible to the observed pixel in every image. Putting the camera poses into the same least-squares problem is bundle adjustment (Barfoot, 2nd ed., Section 10.1), where note 5 picks up.

When one homography is enough

HZ eq. 9.7 writes the second image's pixel as two terms: $x'=K'RK^{-1}x+K't/Z$. The first term depends only on rotation and intrinsics; the second changes with depth $Z$. Which term vanishes, or becomes predictable, decides which relation you need:

The ground-plane ranging above uses the first row: points on the ground map one-to-one to pixels, and the mapping breaks as soon as a point leaves the ground.

As of October 2026: the geometry has not aged, only who computes it

In 2025–2026, feed-forward 3D models output camera intrinsics and extrinsics, depth and pointmaps in a single forward pass. VGGT won the CVPR 2025 Best Paper, and its authors report reconstruction in under one second; the CVPR 2026 Best Paper went to D4RT, which reconstructs dynamic scenes in 4D (CVPR 2026 closing page). What these models output is still this note's $K$, $[R\mid t]$ and depth; reading, checking and using those outputs still takes this note's geometry. Details are in note 4.

This layer is also something public job descriptions spell out: of the 13 public job descriptions for cars and robots we read on 2026-10-15, about 5 name 3D geometric vision, SLAM/VIO, sensor fusion or BEV, and some ask for the pinhole camera model and epipolar geometry by name (an aggregate of knowledge areas only; no company is named).

Further reading: the HZ sample chapters (chapter 9 on epipolar geometry is free), Szeliski, 2nd ed. (free download for personal use), and the course pages of CMU 16-385 and 16-822. Next: state estimation and tracking.