{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14809","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14809/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14809/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14809/events","html_url":"https://github.com/tensorflow/tensorflow/issues/14809","id":276194025,"number":14809,"title":"Batch Normalization layer is unusable","user":{"login":"nikonikolov","id":11044035,"avatar_url":"https://avatars3.githubusercontent.com/u/11044035?v=4","gravatar_id":"","url":"https://api.github.com/users/nikonikolov","html_url":"https://github.com/nikonikolov","followers_url":"https://api.github.com/users/nikonikolov/followers","following_url":"https://api.github.com/users/nikonikolov/following{/other_user}","gists_url":"https://api.github.com/users/nikonikolov/gists{/gist_id}","starred_url":"https://api.github.com/users/nikonikolov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nikonikolov/subscriptions","organizations_url":"https://api.github.com/users/nikonikolov/orgs","repos_url":"https://api.github.com/users/nikonikolov/repos","events_url":"https://api.github.com/users/nikonikolov/events{/privacy}","received_events_url":"https://api.github.com/users/nikonikolov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-11-22T20:21:21Z","updated_at":"2017-11-23T11:41:40Z","closed_at":null,"author_association":"NONE","body":"Despite the numerous submitted issues, `tf.layers.batch_normalization` still feels completely unusable. The major problems are:\r\n\r\n1. It does not allow for input tensors with varying shapes. It is complete nonsense to have a fixed batch size. It should be allowed for the batch dimension to be vary.\r\n\r\n2. One needs to manually update the running mean and variance. This is very uncomfortable and a very common pitfall for many beginners, while it would take just a couple of lines to do the update internally based on the value of the `training` parameter.\r\n\r\nI have recently seen too many custom implementations of a batch normalization layer because of the above problems and it will definitely be very useful if these problems are fixed ASAP.\r\n\r\nI am using `tensorflow-gpu`, version `1.4`"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/346596454","html_url":"https://github.com/tensorflow/tensorflow/issues/14809#issuecomment-346596454","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14809","id":346596454,"user":{"login":"nikonikolov","id":11044035,"avatar_url":"https://avatars3.githubusercontent.com/u/11044035?v=4","gravatar_id":"","url":"https://api.github.com/users/nikonikolov","html_url":"https://github.com/nikonikolov","followers_url":"https://api.github.com/users/nikonikolov/followers","following_url":"https://api.github.com/users/nikonikolov/following{/other_user}","gists_url":"https://api.github.com/users/nikonikolov/gists{/gist_id}","starred_url":"https://api.github.com/users/nikonikolov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nikonikolov/subscriptions","organizations_url":"https://api.github.com/users/nikonikolov/orgs","repos_url":"https://api.github.com/users/nikonikolov/repos","events_url":"https://api.github.com/users/nikonikolov/events{/privacy}","received_events_url":"https://api.github.com/users/nikonikolov/received_events","type":"User","site_admin":false},"created_at":"2017-11-23T11:41:40Z","updated_at":"2017-11-23T11:41:40Z","author_association":"NONE","body":"Thanks for the quick reply!\r\n\r\n1. It does not seem to work for me. \r\n```\r\n>>> data = tf.placeholder(tf.float32, shape=[None, 2, 3])\r\n>>> norm = tf.layers.batch_normalization(data, axis=0)\r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 1, in <module>\r\n  File \"/usr/lib64/python3.6/site-packages/tensorflow/python/layers/normalization.py\", line 586, in batch_normalization\r\n    return layer.apply(inputs, training=training)\r\n  File \"/usr/lib64/python3.6/site-packages/tensorflow/python/layers/base.py\", line 671, in apply\r\n    return self.__call__(inputs, *args, **kwargs)\r\n  File \"/usr/lib64/python3.6/site-packages/tensorflow/python/layers/base.py\", line 559, in __call__\r\n    self.build(input_shapes[0])\r\n  File \"/usr/lib64/python3.6/site-packages/tensorflow/python/layers/normalization.py\", line 179, in build\r\n    input_shape)\r\nValueError: ('Input has undefined `axis` dimension. Input shape: ', TensorShape([Dimension(None), Dimension(2), Dimension(3)]))\r\n```\r\nIn the source code, it seems that the batch dimension must be fixed: https://github.com/tensorflow/tensorflow/blob/b5df90f91cde6eb12af9cbe818bd2cf4a9bcc687/tensorflow/python/layers/normalization.py#L176-L179 Am I doing something wrong?\r\n\r\n\r\n2. Out of curiosity, is it slower because computing the input for the next layer takes more time when you do the update internally? In any case, there is often a trade-off between code readability and performance, so in my opinion it would definitely be much better if there is some parameter that controls if the update is done internally or not, rather than leaving no choice for the programmer. It would also be a good, if possible, to provide some approximate metric of the speed up, when the update is not done internally, so the programmer can choose for himself. "}}
{"actor":{"display_login":"nikonikolov"}}