Multi-View Stereo from Keyframe Depth and Optical Flow

Classical multi-view stereo pipelines recover camera pose through optimization, using PnP with RANSAC followed by bundle adjustment. This is accurate, but slow, and it only happens after the network has already run. More recent models like DUSt3R and VGGT predict geometry directly, but they need large amounts of data and compute to train, which can be a barrier for many people.

In my thesis, I extended DUSt3R in two ways to address both of these issues. First, instead of RGB images, the model takes a depth estimate of the keyframe and an optical flow estimate of the second frame. Second, along with the point maps, it predicts a bundle of camera rays in Plücker coordinates, from which the relative camera pose can be recovered in closed form.

DUSt3R takes two images and predicts a point map for each, which assigns every pixel a 3D point in the coordinate frame of the first camera. Both images are encoded into tokens, which then pass through a decoder where each block applies self-attention within an image followed by cross-attention to the other image, so information is constantly shared between the two branches. A regression head then turns each branch's tokens into a point map and a confidence map.

The architecture has no explicit geometric inductive biases, so it relies on data to learn priors for depth, pose and intrinsics, which is why these models need so much training data.

The original model encodes both images with a shared encoder, which is not enough when the two inputs are different modalities, so I use a separate encoder for each. The depth and flow estimates are precomputed with Depth Anything V2 and DPFlow. These inputs describe the geometry of the scene rather than its appearance, so the priors the model learns do not depend on RGB and generalize better to unseen data.

I also found that computing the flow in the reverse direction, from the second frame to the first, gave much better results than the forward direction, since it includes information about pixels in the second image that are not visible in the first. The downside of optical flow is that it assumes brightness consistency, which only holds for a narrow baseline, so the model does not work on sparse, widely spaced views.

To predict the pose of the second camera, I represent it as a bundle of rays, one through each pixel, written in Plücker coordinates. The Plücker coordinates of a line are ⟨d, m⟩, where d is its direction and m = p × d is its moment for any point p on the line. The moment is the same whichever point on the line is chosen, so each ray is a 6-vector, and an extra DPT head predicts a 6-channel ray map for the second image from the tokens of both decoders.

For supervision, the ground truth rays come from the known intrinsics K and camera pose R, t. Every ray passes through the camera centre, so using it as p gives d = Rᵀ K⁻¹ u and m = (−Rᵀ t) × d for a pixel u.

The first camera is the world frame, so the translation of the second camera is the position of its centre o. Every ray passes through this point, so m = o × d holds for every pixel, which is linear in o and can be solved by least squares over the predicted rays. Solving over every pixel was expensive in the backward pass, so I use a random subset of pixels instead.

For the rotation, I use the Kabsch algorithm to find the rotation that best aligns the ray directions of the first camera with the ones predicted for the second. Both steps are closed form, so there is no PnP, RANSAC or bundle adjustment after the network runs.

I train on roughly 100k image pairs from BlendedMVS, a synthetic dataset of mostly outdoor scenes, which is about 1% of the data used by DUSt3R and VGGT. Pairs with very wide baselines are filtered out, since the flow estimates fail on them. As in DUSt3R, the model is first trained on 224×224 images and then on 512×512, using a confidence-weighted 3D regression loss on the point maps.

The camera head is then trained separately, with the encoders, decoders and point-map heads frozen. Its loss is CamLoss = RRE + RTE + MSE. The relative rotation and translation errors are computed on the recovered pose, and neither depends on the scale of the scene. In practice, I noticed that the predicted coordinates grew very large after a few epochs without the MSE term, so it acts as a regulariser.

The model works on pairs, so for a video I choose a keyframe and predict every other frame against it, then transform everything back to the frame of the first camera. It performs best when the keyframe is near the middle of the sequence. Longer sequences can be split across several overlapping keyframes with their poses chained together, so inference time grows linearly with the number of frames. The trade-off is that errors accumulate along the chain.

Results on 7-Scenes, a dataset of real indoor scenes that the model never saw during training.

While the model does not beat the baselines on these metrics, it is competitive across the board and is the fastest end-to-end, since it recovers pose without any post-processing.