{"action":"created","issue":{"url":"https://api.github.com/repos/BVLC/caffe/issues/4885","repository_url":"https://api.github.com/repos/BVLC/caffe","labels_url":"https://api.github.com/repos/BVLC/caffe/issues/4885/labels{/name}","comments_url":"https://api.github.com/repos/BVLC/caffe/issues/4885/comments","events_url":"https://api.github.com/repos/BVLC/caffe/issues/4885/events","html_url":"https://github.com/BVLC/caffe/issues/4885","id":184861632,"number":4885,"title":"During test phase, the output of Batch Normalization layer is not equal to (input-mean)/sqrt(var + eps)?","user":{"login":"XuefeiW","id":15201418,"avatar_url":"https://avatars1.githubusercontent.com/u/15201418?v=4","gravatar_id":"","url":"https://api.github.com/users/XuefeiW","html_url":"https://github.com/XuefeiW","followers_url":"https://api.github.com/users/XuefeiW/followers","following_url":"https://api.github.com/users/XuefeiW/following{/other_user}","gists_url":"https://api.github.com/users/XuefeiW/gists{/gist_id}","starred_url":"https://api.github.com/users/XuefeiW/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/XuefeiW/subscriptions","organizations_url":"https://api.github.com/users/XuefeiW/orgs","repos_url":"https://api.github.com/users/XuefeiW/repos","events_url":"https://api.github.com/users/XuefeiW/events{/privacy}","received_events_url":"https://api.github.com/users/XuefeiW/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-10-24T14:45:45Z","updated_at":"2017-08-04T19:48:51Z","closed_at":"2016-10-25T09:08:03Z","body":"I am trying to convert a pre-trained ResNet-50 from Caffe to Torch. I've built the structure in Torch and copied the parameters from Caffe to Torch. When I compare the layer output, after first batch normalization layer, the output starts becoming different. I notice BN layer implementation in Caffe does not include a factor and bias (which is implemented in scale layer), so I tried calculating the normalization part independently by (inputData - mean)/sqrt(variance + eps), but the BN layer output in Caffe is different to this as well. \n\nHere is the first two layers prototxt:\nlayer {\n  name: \"conv_1\"\n  type: \"Convolution\"\n  bottom: \"data\"\n  top: \"conv_1\"\n  param {\n    lr_mult: 1\n    decay_mult: 1\n  }\n  param {\n    lr_mult: 2\n    decay_mult: 0\n  }\n  convolution_param {\n    num_output: 64\n    pad: 3\n    kernel_size: 7\n    stride: 2\n    weight_filler {\n      type: \"xavier\"\n    }\n    bias_filler {\n      type: \"constant\"\n      value: 0\n    }\n  }\n}\nlayer {\n  name: \"bn_1\"\n  type: \"BatchNorm\"\n  bottom: \"conv_1\"\n  top: \"conv_1\"\n  param {\n    lr_mult: 0\n    decay_mult: 0\n  }\n  param {\n    lr_mult: 0\n    decay_mult: 0\n  }\n  param {\n    lr_mult: 0\n    decay_mult: 0\n  }\n  batch_norm_param {\n    use_global_stats: true\n  }\n}\n\nHere is how I get parameters from BN layer:\nnet.params['bn_1'][0].data    ---  mean \nfirst element: -1345.0353            --- mean for first channel\nnet.params['bn_1'][1].data    ---  variance \nfirst element: 1692061.5            --- variance for first channel\nnet.params['bn_1'][2].data    --- not sure this one is eps or moving average factor\nfirst and only element: 999.98236084\n\nThe input of BN layer is the output of Convolution layer, which I calculate by this:\nimg = np.full((1,3,224,224), 0.5)  -- define a matrix with all elements are 0.5\nnet.blobs['conv_1'].data[...] = img\noutput = net.forward(start='data', end='conv_1')\n\nThe first element of convolution layer output is: -0.340065956\n\nSo I expect the first element of BN layer output should be: \n                 (-0.340065956 - (-1345.0353))/sqrt(1692061.5) = 1.03375\nBut the first element of BN layer output is : 0.02443156\ncode: net.forward(start='data', end='bn_1')\nThere is a huge difference and a small eps won't change the result from 1.03375 to 0.02443.\n\nI am not sure whether I missed some calculation or I did some miscalculation. Could anyone help me with that? Thank you!\n"},"comment":{"url":"https://api.github.com/repos/BVLC/caffe/issues/comments/320337925","html_url":"https://github.com/BVLC/caffe/issues/4885#issuecomment-320337925","issue_url":"https://api.github.com/repos/BVLC/caffe/issues/4885","id":320337925,"user":{"login":"extragoya","id":12091343,"avatar_url":"https://avatars0.githubusercontent.com/u/12091343?v=4","gravatar_id":"","url":"https://api.github.com/users/extragoya","html_url":"https://github.com/extragoya","followers_url":"https://api.github.com/users/extragoya/followers","following_url":"https://api.github.com/users/extragoya/following{/other_user}","gists_url":"https://api.github.com/users/extragoya/gists{/gist_id}","starred_url":"https://api.github.com/users/extragoya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/extragoya/subscriptions","organizations_url":"https://api.github.com/users/extragoya/orgs","repos_url":"https://api.github.com/users/extragoya/repos","events_url":"https://api.github.com/users/extragoya/events{/privacy}","received_events_url":"https://api.github.com/users/extragoya/received_events","type":"User","site_admin":false},"created_at":"2017-08-04T19:48:51Z","updated_at":"2017-08-04T19:48:51Z","body":"Thank you @XuefeiW . One question I have is that I observe the third blob to always have the same value of: 999.98236084 (which is also reported on your comment), on different networks, different datasets, and different iteration numbers. My understanding is that the third blob should hold the (roughly) the number of batch norm iterations, weighted appropriately by the moving_average_fraction. Do you also observe having 999.98236084, or a number close to it, all the time?  "}}
{"actor":{"display_login":"extragoya"}}