{"action":"created","issue":{"url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/46","repository_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark","labels_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/46/labels{/name}","comments_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/46/comments","events_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/46/events","html_url":"https://github.com/yahoo/TensorFlowOnSpark/issues/46","id":212387107,"number":46,"title":"Slim throws all Variables must have their device set: name: \"global_step\" after upgrade tensorflow to v1.0.0 ","user":{"login":"wangyum","id":5399861,"avatar_url":"https://avatars3.githubusercontent.com/u/5399861?v=3","gravatar_id":"","url":"https://api.github.com/users/wangyum","html_url":"https://github.com/wangyum","followers_url":"https://api.github.com/users/wangyum/followers","following_url":"https://api.github.com/users/wangyum/following{/other_user}","gists_url":"https://api.github.com/users/wangyum/gists{/gist_id}","starred_url":"https://api.github.com/users/wangyum/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wangyum/subscriptions","organizations_url":"https://api.github.com/users/wangyum/orgs","repos_url":"https://api.github.com/users/wangyum/repos","events_url":"https://api.github.com/users/wangyum/events{/privacy}","received_events_url":"https://api.github.com/users/wangyum/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-03-07T10:15:20Z","updated_at":"2017-04-26T14:05:11Z","closed_at":null,"body":"\r\nWhen running [slim](https://github.com/yahoo/TensorFlowOnSpark/tree/master/examples/slim) example, Tensorflow v0.12.0 is OK, but Tensorflow v1.0.0 will throw exception: \r\n\r\n```\r\norg.apache.spark.api.python.PythonException: Traceback (most recent call last):\r\n  File \"/opt/cloudera/parcels/YSPARK-2.2.0-2.6.0.d20170305-11.04.04-success_flag-6b0cfd9fa51aca4536d7c3f2a4bbceae11a50339/lib/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 180, in main\r\n    process()\r\n  File \"/opt/cloudera/parcels/YSPARK-2.2.0-2.6.0.d20170305-11.04.04-success_flag-6b0cfd9fa51aca4536d7c3f2a4bbceae11a50339/lib/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 175, in process\r\n    serializer.dump_stream(func(split_index, iterator), outfile)\r\n  File \"/opt/cloudera/parcels/YSPARK-2.2.0-2.6.0.d20170305-11.04.04-success_flag-6b0cfd9fa51aca4536d7c3f2a4bbceae11a50339/lib/spark/python/lib/pyspark.zip/pyspark/rdd.py\", line 2407, in pipeline_func\r\n  File \"/opt/cloudera/parcels/YSPARK-2.2.0-2.6.0.d20170305-11.04.04-success_flag-6b0cfd9fa51aca4536d7c3f2a4bbceae11a50339/lib/spark/python/lib/pyspark.zip/pyspark/rdd.py\", line 2407, in pipeline_func\r\n  File \"/opt/cloudera/parcels/YSPARK-2.2.0-2.6.0.d20170305-11.04.04-success_flag-6b0cfd9fa51aca4536d7c3f2a4bbceae11a50339/lib/spark/python/lib/pyspark.zip/pyspark/rdd.py\", line 2407, in pipeline_func\r\n  File \"/opt/cloudera/parcels/YSPARK-2.2.0-2.6.0.d20170305-11.04.04-success_flag-6b0cfd9fa51aca4536d7c3f2a4bbceae11a50339/lib/spark/python/lib/pyspark.zip/pyspark/rdd.py\", line 346, in func\r\n  File \"/opt/cloudera/parcels/YSPARK-2.2.0-2.6.0.d20170305-11.04.04-success_flag-6b0cfd9fa51aca4536d7c3f2a4bbceae11a50339/lib/spark/python/lib/pyspark.zip/pyspark/rdd.py\", line 794, in func\r\n  File \"/data/M00/tandem/wangyuming/TensorFlowOnSpark/tfspark.zip/com/yahoo/ml/tf/TFSparkNode.py\", line 235, in _mapfn\r\n  File \"/data/M00/tandem/wangyuming/TensorFlowOnSpark/examples/slim/train_image_classifier.py\", line 603, in main_fun\r\n  File \"/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/learning.py\", line 715, in train\r\n    init_fn=init_fn)\r\n  File \"/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py\", line 336, in __init__\r\n    self._verify_setup()\r\n  File \"/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py\", line 881, in _verify_setup\r\n    \"their device set: %s\" % op)\r\nValueError: When using replicas, all Variables must have their device set: name: \"global_step\"\r\nop: \"VariableV2\"\r\nattr {\r\n  key: \"_class\"\r\n  value {\r\n    list {\r\n      s: \"loc:@global_step\"\r\n    }\r\n  }\r\n}\r\nattr {\r\n  key: \"container\"\r\n  value {\r\n    s: \"\"\r\n  }\r\n}\r\nattr {\r\n  key: \"dtype\"\r\n  value {\r\n    type: DT_INT64\r\n  }\r\n}\r\nattr {\r\n  key: \"shape\"\r\n  value {\r\n    shape {\r\n    }\r\n  }\r\n}\r\nattr {\r\n  key: \"shared_name\"\r\n  value {\r\n    s: \"\"\r\n  }\r\n}\r\n\r\n\r\n\tat org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)\r\n\tat org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)\r\n\tat org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)\r\n\tat org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:108)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:317)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n```\r\nTo run slim on Tensorflow v1.0.0, I use [tf_upgrade.py](https://github.com/tensorflow/tensorflow/blob/v1.0.0/tensorflow/tools/compatibility/tf_upgrade.py) update some code:\r\nhttps://github.com/wangyum/TensorFlowOnSpark/tree/slim-tensorflow-1.0"},"comment":{"url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/comments/297419007","html_url":"https://github.com/yahoo/TensorFlowOnSpark/issues/46#issuecomment-297419007","issue_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/46","id":297419007,"user":{"login":"tobiajo","id":9915543,"avatar_url":"https://avatars1.githubusercontent.com/u/9915543?v=3","gravatar_id":"","url":"https://api.github.com/users/tobiajo","html_url":"https://github.com/tobiajo","followers_url":"https://api.github.com/users/tobiajo/followers","following_url":"https://api.github.com/users/tobiajo/following{/other_user}","gists_url":"https://api.github.com/users/tobiajo/gists{/gist_id}","starred_url":"https://api.github.com/users/tobiajo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tobiajo/subscriptions","organizations_url":"https://api.github.com/users/tobiajo/orgs","repos_url":"https://api.github.com/users/tobiajo/repos","events_url":"https://api.github.com/users/tobiajo/events{/privacy}","received_events_url":"https://api.github.com/users/tobiajo/received_events","type":"User","site_admin":false},"created_at":"2017-04-26T14:05:11Z","updated_at":"2017-04-26T14:05:11Z","body":"Any updates regarding this issue? I am trying to run multi node Slim.\r\n\r\nI tried:\r\n'global_step = tf.identity(slim.create_global_step(), name=\"global_step\")'\r\n\r\nBut that just resulted in another error."}}
{"actor":{"display_login":"tobiajo"}}