An Empirical Study of Batch Normalization and Group Normalization in
  Conditional Computation
  Batch normalization has been widely used to improve optimization in deep
neural networks. While the uncertainty in batch statistics can act as a
regularizer, using these dataset statistics specific to the training set
impairs generalization in certain tasks. Recently, alternative methods for
normalizing feature activations in neural networks have been proposed. Among
them, group normalization has been shown to yield similar, in some domains even
superior performance to batch normalization. All these methods utilize a
learned affine transformation after the normalization operation to increase
representational power. Methods used in conditional computation define the
parameters of these transformations as learnable functions of conditioning
information. In this work, we study whether and where the conditional
formulation of group normalization can improve generalization compared to
conditional batch normalization. We evaluate performances on the tasks of
visual question answering, few-shot learning, and conditional image generation.
