Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views
  of Novel Scenes
  Recent neural view synthesis methods have achieved impressive quality and
realism, surpassing classical pipelines which rely on multi-view
reconstruction. State-of-the-Art methods, such as NeRF, are designed to learn a
single scene with a neural network and require dense multi-view inputs. Testing
on a new scene requires re-training from scratch, which takes 2-3 days. In this
work, we introduce Stereo Radiance Fields (SRF), a neural view synthesis
approach that is trained end-to-end, generalizes to new scenes, and requires
only sparse views at test time. The core idea is a neural architecture inspired
by classical multi-view stereo methods, which estimates surface points by
finding similar image regions in stereo images. In SRF, we predict color and
density for each 3D point given an encoding of its stereo correspondence in the
input images. The encoding is implicitly learned by an ensemble of pair-wise
similarities -- emulating classical stereo. Experiments show that SRF learns
structure instead of overfitting on a scene. We train on multiple scenes of the
DTU dataset and generalize to new ones without re-training, requiring only 10
sparse and spread-out views as input. We show that 10-15 minutes of fine-tuning
further improve the results, achieving significantly sharper, more detailed
results than scene-specific models. The code, model, and videos are available
at https://virtualhumans.mpi-inf.mpg.de/srf/.
