HDR Environment Map Estimation for Real-Time Augmented Reality
  We present a method to estimate an HDR environment map from a narrow
field-of-view LDR camera image in real-time. This enables perceptually
appealing reflections and shading on virtual objects of any material finish,
from mirror to diffuse, rendered into a real physical environment using
augmented reality. Our method is based on our efficient convolutional neural
network architecture, EnvMapNet, trained end-to-end with two novel losses,
ProjectionLoss for the generated image, and ClusterLoss for adversarial
training. Through qualitative and quantitative comparison to state-of-the-art
methods, we demonstrate that our algorithm reduces the directional error of
estimated light sources by more than 50%, and achieves 3.7 times lower Frechet
Inception Distance (FID). We further showcase a mobile application that is able
to run our neural network model in under 9 ms on an iPhone XS, and render in
real-time, visually coherent virtual objects in previously unseen real-world
environments.
