Planar Geometry with a Single Camera

A camera maps the world by x ~ K [R | t] X. For points on a plane, we can choose coordinates so that the plane is z = 0. The third column of R then multiplies zero, and the projection reduces to a 3×3 homography, which the rest of this project builds on.

Estimating the homography from correspondences gives us stitching and rectification. If K is known, we can extract [r1 r2 t] from it to get the pose, since two orthonormal columns determine the third. If K is unknown, we can instead constrain it using this orthonormality across many views, which gives us calibration.

I only use OpenCV and imageio to read and write images, so there are no calls to findHomography, warpPerspective, solvePnP, calibrateCamera, cornerHarris or matchTemplate. I implemented the SVD-based DLT solve, Hartley normalisation, the RANSAC loop with its adaptive iteration count, the Harris response, the descriptors and ratio test, bilinear resampling and feathered blending, the pose decomposition, the NCC tracker and the Cholesky factorisation of the absolute conic myself.

A homography has eight degrees of freedom, and each point correspondence gives two equations, so four correspondences are enough to determine it. With more, the DLT stacks all of the equations into one matrix and takes the singular vector with the smallest singular value as the solution.

Pixel coordinates are in the hundreds while the homogeneous coordinate is 1, so this matrix is badly conditioned, and the error grows quickly with noise. Hartley normalisation fixes this by translating the points to have zero mean and scaling them to an average distance of √2 from the origin before solving.

Instead of clicking correspondences by hand, I detect Harris corners in both images and describe each one with a small patch, normalised to zero mean and unit norm. The dot product of two such patches is their normalised cross-correlation, which makes the matching insensitive to brightness and contrast. A match is only kept if it passes Lowe's ratio test and is mutual.

For the panorama, each image is warped into a common frame by backward warping with bilinear resampling. The overlapping images are then blended by weighting each one by its distance from its own boundary, which hides the seam.

Some of the matches will always be wrong, and a single bad match is enough to ruin a least squares fit. RANSAC repeatedly fits a homography to four random correspondences, counts how many of the others agree with it, and keeps the hypothesis with the most inliers. The number of iterations is updated from the best inlier rate found so far, N = log(1-p) / log(1-w⁴), so clean data needs very few iterations.

Rectification applies a homography to undo perspective. Given the four corners of a planar object with a known aspect ratio, such as a number plate, we can map it to a fronto-parallel view in which distances are metric.

With known intrinsics, K⁻¹H should equal [r1 r2 t] up to scale, so its first two columns give two axes of the rotation and their cross product gives the third. Since the estimate is noisy, I then project the result onto the nearest rotation matrix.

The plane is tracked through the video by finding each of its four corners with normalised cross-correlation in every frame. Each frame is matched against the original templates rather than the previous frame, so the track does not drift over time.

Zhang's method removes the need to know K in advance. The columns r1 and r2 must be orthogonal and of equal length, which gives two linear constraints per view on ω = K⁻ᵀK⁻¹, the image of the absolute conic. With enough views, ω can be solved for, and K recovered from it by Cholesky factorisation.

This only works well when the plane is seen at a range of different tilts, since views at similar orientations give nearly the same constraints, and the solution then becomes very sensitive to noise in the corners.