Deep FisherNet for Object Classification
  Despite the great success of convolutional neural networks (CNN) for the
image classification task on datasets like Cifar and ImageNet, CNN's
representation power is still somewhat limited in dealing with object images
that have large variation in size and clutter, where Fisher Vector (FV) has
shown to be an effective encoding strategy. FV encodes an image by aggregating
local descriptors with a universal generative Gaussian Mixture Model (GMM). FV
however has limited learning capability and its parameters are mostly fixed
after constructing the codebook. To combine together the best of the two
worlds, we propose in this paper a neural network structure with FV layer being
part of an end-to-end trainable system that is differentiable; we name our
network FisherNet that is learnable using backpropagation. Our proposed
FisherNet combines convolutional neural network training and Fisher Vector
encoding in a single end-to-end structure. We observe a clear advantage of
FisherNet over plain CNN and standard FV in terms of both classification
accuracy and computational efficiency on the challenging PASCAL VOC object
classification task.
