Learning Sensor Multiplexing Design through Back-propagation
  Recent progress on many imaging and vision tasks has been driven by the use
of deep feed-forward neural networks, which are trained by propagating
gradients of a loss defined on the final output, back through the network up to
the first layer that operates directly on the image. We propose
back-propagating one step further---to learn camera sensor designs jointly with
networks that carry out inference on the images they capture. In this paper, we
specifically consider the design and inference problems in a typical color
camera---where the sensor is able to measure only one color channel at each
pixel location, and computational inference is required to reconstruct a full
color image. We learn the camera sensor's color multiplexing pattern by
encoding it as layer whose learnable weights determine which color channel,
from among a fixed set, will be measured at each location. These weights are
jointly trained with those of a reconstruction network that operates on the
corresponding sensor measurements to produce a full color image. Our network
achieves significant improvements in accuracy over the traditional Bayer
pattern used in most color cameras. It automatically learns to employ a sparse
color measurement approach similar to that of a recent design, and moreover,
improves upon that design by learning an optimal layout for these measurements.
