Pixel-Adaptive Convolutional Neural Networks
  Convolutions are the fundamental building block of CNNs. The fact that their
weights are spatially shared is one of the main reasons for their widespread
use, but it also is a major limitation, as it makes convolutions content
agnostic. We propose a pixel-adaptive convolution (PAC) operation, a simple yet
effective modification of standard convolutions, in which the filter weights
are multiplied with a spatially-varying kernel that depends on learnable, local
pixel features. PAC is a generalization of several popular filtering techniques
and thus can be used for a wide range of use cases. Specifically, we
demonstrate state-of-the-art performance when PAC is used for deep joint image
upsampling. PAC also offers an effective alternative to fully-connected CRF
(Full-CRF), called PAC-CRF, which performs competitively, while being
considerably faster. In addition, we also demonstrate that PAC can be used as a
drop-in replacement for convolution layers in pre-trained networks, resulting
in consistent performance improvements.
