Learning from Shader Program Traces
  Deep learning for image processing typically treats input imagery as pixels
in some color space. This paper proposes instead to learn from program traces
of procedural fragment shaders -- programs that generate images. At each pixel,
we collect the intermediate values computed at program execution, and these
data form the input to the learned model. We investigate this learning task for
a variety of applications: our model can learn to predict a low-noise output
image from shader programs that exhibit sampling noise; this model can also
learn from a simplified shader program that approximates the reference solution
with less computation, as well as learn the output of postprocessing filters
like defocus blur and edge-aware sharpening. Finally we show that the idea of
learning from program traces can even be applied to non-imagery simulations of
flocks of boids. Our experiments on a variety of shaders show quantitatively
and qualitatively that models learned from program traces outperform baseline
models learned from RGB color augmented with hand-picked shader-specific
features like normals, depth, and diffuse and specular color.
