{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14353","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14353/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14353/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14353/events","html_url":"https://github.com/tensorflow/tensorflow/issues/14353","id":272109585,"number":14353,"title":"variance goes negative when set layers.batch_norm reuse=True","user":{"login":"coderXiangLi","id":8342490,"avatar_url":"https://avatars3.githubusercontent.com/u/8342490?v=4","gravatar_id":"","url":"https://api.github.com/users/coderXiangLi","html_url":"https://github.com/coderXiangLi","followers_url":"https://api.github.com/users/coderXiangLi/followers","following_url":"https://api.github.com/users/coderXiangLi/following{/other_user}","gists_url":"https://api.github.com/users/coderXiangLi/gists{/gist_id}","starred_url":"https://api.github.com/users/coderXiangLi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coderXiangLi/subscriptions","organizations_url":"https://api.github.com/users/coderXiangLi/orgs","repos_url":"https://api.github.com/users/coderXiangLi/repos","events_url":"https://api.github.com/users/coderXiangLi/events{/privacy}","received_events_url":"https://api.github.com/users/coderXiangLi/received_events","type":"User","site_admin":false},"labels":[{"id":404586594,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20tensorflower","name":"stat:awaiting tensorflower","color":"b60205","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-11-08T08:21:05Z","updated_at":"2017-12-20T19:18:56Z","closed_at":null,"author_association":"NONE","body":"#### TF 1.2.1 running on cpu & distributed version\r\n\r\n#### layers.batch_norm set reuse=True\r\nI use a sequence of items features which contains n (feature length) * N (sequence length) real-value features. And do batch_norm on each item of the sequence, then do some full_connected, etc. Finally concat them as dnn input.\r\nHere is a simple code of the this. In order to make variance quickly go negative, I set decay=0.6.\r\n\r\n\timport tensorflow as tf\r\n\tfrom tensorflow.contrib import layers\r\n\timport numpy as np\r\n\t\r\n\tseq_length = 1000\r\n\tbatch = 100\r\n\tlength = 3\r\n\tplace_holders = []\r\n\tseq_raw_f = []\r\n\tfor i in range(seq_length):\r\n\t  x = True if i != 0 else None\r\n\t  sequence_place_holder = tf.placeholder(dtype=tf.float32, shape=[None, length])\r\n\t  place_holders.append(sequence_place_holder)\r\n\t  sequence = layers.batch_norm(inputs=sequence_place_holder, scope=\"bn\", reuse=x, decay=0.6, scale=True,\r\n\t                               # updates_collections=None,\r\n\t                               zero_debias_moving_mean=True)\r\n\t  seq_raw_f.append(sequence)\r\n\t\r\n\tinput = tf.concat(seq_raw_f, axis=0)\r\n\t\r\n\tupdate_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\r\n\t\r\n\tsess = tf.Session()\r\n\tsess.run(tf.initialize_all_variables())\r\n\t\r\n\tplace_holder = {}\r\n\tfor i in range(seq_length):\r\n\t  place_holder[place_holders[i]] = np.random.rand(batch, length)\r\n\t\r\n\tfor i in range(10000):\r\n\t  sess.run(update_ops, place_holder)\r\n\t  # sess.run(input, place_holder)\r\n\t  print sess.run(\"bn/moving_variance:0\")\r\n\t  \r\n#### variance can be :\r\n\r\n\t[ 0.08063364  0.08680637  0.08229597]\r\n\t[ 0.09141719  0.08313672  0.08208766]\r\n\t[ 0.07279671  0.08088712  0.08174741]\r\n\t[-0.1192904  -0.13598624 -0.31083935]\r\n\t[ 0.24966593  0.26548591  0.16420737]\r\n\t[ 0.07931737  0.07920683  0.0833094 ]\r\n\t[ 0.08559028  0.08299339  0.08146621]\r\n\t[ 0.08117087  0.08168832  0.08265808]\r\n\t\r\n\r\n#### Reason probably the code below. \r\nWhen reuse=True, this code will subtract update_delta many times according to reuse times. Then the formula goes wrong. `a * m_v - (1-a) v -> a * m_v -N * (1-a) v.`\r\n\t\r\n\tupdate_delta = (variable - value) * decay\r\n\treturn state_ops.assign_sub(variable, update_delta, name=scope)\r\n\t\r\nwhen, updates_collections=None,  to force update variance, it still happened.\r\n\r\n#### temporary fix\r\n     \r\n     if not zero_debias:\r\n       # variable * decay + value * (1 - decay)\r\n       state_ops.assign(variable, variable * decay + value * (1 - decay))\r\n\r\n\t\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/353157342","html_url":"https://github.com/tensorflow/tensorflow/issues/14353#issuecomment-353157342","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14353","id":353157342,"user":{"login":"tensorflowbutler","id":28546240,"avatar_url":"https://avatars2.githubusercontent.com/u/28546240?v=4","gravatar_id":"","url":"https://api.github.com/users/tensorflowbutler","html_url":"https://github.com/tensorflowbutler","followers_url":"https://api.github.com/users/tensorflowbutler/followers","following_url":"https://api.github.com/users/tensorflowbutler/following{/other_user}","gists_url":"https://api.github.com/users/tensorflowbutler/gists{/gist_id}","starred_url":"https://api.github.com/users/tensorflowbutler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tensorflowbutler/subscriptions","organizations_url":"https://api.github.com/users/tensorflowbutler/orgs","repos_url":"https://api.github.com/users/tensorflowbutler/repos","events_url":"https://api.github.com/users/tensorflowbutler/events{/privacy}","received_events_url":"https://api.github.com/users/tensorflowbutler/received_events","type":"User","site_admin":false},"created_at":"2017-12-20T19:18:56Z","updated_at":"2017-12-20T19:18:56Z","author_association":"MEMBER","body":"It has been 14 days with no activity and the `awaiting tensorflower` label was assigned. Please update the label and/or status accordingly."}}
{"actor":{"display_login":"tensorflowbutler"}}