title: How should "BatchNorm" layer be used in caffe?
body: <p>I am a little confused about how should I use/insert <a href="https://stackoverflow.com/q/41269570/1714410"><code>"BatchNorm"</code></a> layer in my models.<br>
  I see several different approaches, for instance:</p>
  
  <h3><a href="https://github.com/KaimingHe/deep-residual-networks/blob/master/prototxt/ResNet-101-deploy.prototxt#L75-L93" rel="noreferrer">ResNets</a>: <code>"BatchNorm"</code>+<code>"Scale"</code> (no parameter sharing)</h3>
  
  <p><code>"BatchNorm"</code> layer is followed immediately with <code>"Scale"</code> layer: </p>
  
  <pre><code>layer {
      bottom: "res2a_branch1"
      top: "res2a_branch1"
      name: "bn2a_branch1"
      type: "BatchNorm"
      batch_norm_param {
          use_global_stats: true
      }
  }
  
  layer {
      bottom: "res2a_branch1"
      top: "res2a_branch1"
      name: "scale2a_branch1"
      type: "Scale"
      scale_param {
          bias_term: true
      }
  }
  </code></pre>
  
  <h3><a href="https://github.com/BVLC/caffe/blob/master/examples/cifar10/cifar10_full_sigmoid_train_test_bn.prototxt#L68-L82" rel="noreferrer">cifar10 example</a>: only <code>"BatchNorm"</code></h3>
  
  <p>In the cifar10 example provided with caffe, <code>"BatchNorm"</code> is used without any <code>"Scale"</code> following it:</p>
  
  <pre><code>layer {
    name: "bn1"
    type: "BatchNorm"
    bottom: "pool1"
    top: "bn1"
    param {
      lr_mult: 0
    }
    param {
      lr_mult: 0
    }
    param {
      lr_mult: 0
    }
  }
  </code></pre>
  
  <h3><a href="https://gist.github.com/ducha-aiki/6457bbd49fea8b7634a7#file-cifar10_full_sigmoid_train_test_bn-prototxt-L73-L114" rel="noreferrer">cifar10</a> Different <code>batch_norm_param</code> for <code>TRAIN</code> and <code>TEST</code></h3>
  
  <p><code>batch_norm_param: use_global_scale</code> is changed between <code>TRAIN</code> and <code>TEST</code> phase:</p>
  
  <pre><code>layer {
    name: "bn1"
    type: "BatchNorm"
    bottom: "pool1"
    top: "bn1"
    batch_norm_param {
      use_global_stats: false
    }
    param {
      lr_mult: 0
    }
    param {
      lr_mult: 0
    }
    param {
      lr_mult: 0
    }
    include {
      phase: TRAIN
    }
  }
  layer {
    name: "bn1"
    type: "BatchNorm"
    bottom: "pool1"
    top: "bn1"
    batch_norm_param {
      use_global_stats: true
    }
    param {
      lr_mult: 0
    }
    param {
      lr_mult: 0
    }
    param {
      lr_mult: 0
    }
    include {
      phase: TEST
    }
  }
  </code></pre>
  
  <h3>So what should it be?</h3>
  
  <p>How should one use<code>"BatchNorm"</code> layer in caffe?</p>