{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/13778","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/13778/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/13778/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/13778/events","html_url":"https://github.com/tensorflow/tensorflow/issues/13778","id":266059878,"number":13778,"title":"program could not run in docker but could run on local machine","user":{"login":"guotong1988","id":4702353,"avatar_url":"https://avatars3.githubusercontent.com/u/4702353?v=4","gravatar_id":"","url":"https://api.github.com/users/guotong1988","html_url":"https://github.com/guotong1988","followers_url":"https://api.github.com/users/guotong1988/followers","following_url":"https://api.github.com/users/guotong1988/following{/other_user}","gists_url":"https://api.github.com/users/guotong1988/gists{/gist_id}","starred_url":"https://api.github.com/users/guotong1988/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guotong1988/subscriptions","organizations_url":"https://api.github.com/users/guotong1988/orgs","repos_url":"https://api.github.com/users/guotong1988/repos","events_url":"https://api.github.com/users/guotong1988/events{/privacy}","received_events_url":"https://api.github.com/users/guotong1988/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-10-17T09:47:26Z","updated_at":"2017-10-19T02:24:28Z","closed_at":"2017-10-19T02:24:28Z","author_association":"CONTRIBUTOR","body":"### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**:\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: \r\nCentOS7 in local machine\r\nUbuntu 16.04.2 LTS in docker image\r\n- **TensorFlow installed from (source or binary)**:\r\n- **TensorFlow version (use command below)**:\r\nv1.3.0-rc0-33-g6f0d70e 1.3.0-rc1\r\n- **Python version**: \r\n3.5.3\r\n- **Bazel version (if compiling from source)**:\r\n- **CUDA/cuDNN version**:\r\n8.0 6.0\r\n- **GPU model and memory**:\r\nM40 24G\r\n\r\n\r\n\r\n### Describe the problem\r\nI could run my program on local machine, but can not run on docker(ubuntu image).\r\n\r\nthe stacktrace of the program in nvidia-docker:\r\n\r\n```\r\n2017-10-17 09:02:50.757982: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758004: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758020: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758052: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758073: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758089: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758134: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758154: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758178: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758209: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758229: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758246: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758289: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758323: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758337: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758464: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758500: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758532: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758562: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758593: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758624: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758851: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758876: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758891: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758922: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758942: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758957: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.758986: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.759006: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.759021: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.759065: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.759085: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.759100: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.759129: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.759149: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.759170: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.759203: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.759223: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n2017-10-17 09:02:50.759283: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\nTraceback (most recent call last):\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 1327, in _do_call\r\n    return fn(*args)\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 1306, in _run_fn\r\n    status, run_metadata)\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/contextlib.py\", line 66, in __exit__\r\n    next(self.gen)\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py\", line 466, in raise_exception_on_not_ok_status\r\n    pywrap_tensorflow.TF_GetCode(status))\r\ntensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n\t [[Node: grads_0/gradients/AddN_2519/_461 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/cpu:0\", send_device=\"/job:localhost/replica:0/task:0/gpu:0\", send_device_incarnation=1, tensor_name=\"edge_289171_grads_0/gradients/AddN_2519\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/cpu:0\"]()]]\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/cli.py\", line 113, in <module>\r\n    tf.app.run()\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/platform/app.py\", line 48, in run\r\n    _sys.exit(main(_sys.argv[:1] + flags_passthrough))\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/cli.py\", line 110, in main\r\n    m(config)\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/main.py\", line 25, in main\r\n    _train(config)\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/main.py\", line 102, in _train\r\n    loss, summary, train_op = trainer.step(sess, batches, get_summary=get_summary)\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/trainer.py\", line 71, in step\r\n    loss, train_op = sess.run([self.loss, self.train_op], feed_dict=feed_dict)\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 895, in run\r\n    run_metadata_ptr)\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 1124, in _run\r\n    feed_dict_tensor, options, run_metadata)\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 1321, in _do_run\r\n    options, run_metadata)\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 1340, in _do_call\r\n    raise type(e)(node_def, op, message)\r\ntensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n\t [[Node: grads_0/gradients/AddN_2519/_461 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/cpu:0\", send_device=\"/job:localhost/replica:0/task:0/gpu:0\", send_device_incarnation=1, tensor_name=\"edge_289171_grads_0/gradients/AddN_2519\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/cpu:0\"]()]]\r\n\r\nCaused by op 'grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs', defined at:\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/cli.py\", line 113, in <module>\r\n    tf.app.run()\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/platform/app.py\", line 48, in run\r\n    _sys.exit(main(_sys.argv[:1] + flags_passthrough))\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/cli.py\", line 110, in main\r\n    m(config)\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/main.py\", line 25, in main\r\n    _train(config)\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/main.py\", line 87, in _train\r\n    trainer = MultiGPUTrainer(config, models)\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/trainer.py\", line 52, in __init__\r\n    grads = self.opt.compute_gradients(loss, var_list=self.var_list)\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py\", line 386, in compute_gradients\r\n    colocate_gradients_with_ops=colocate_gradients_with_ops)\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py\", line 542, in gradients\r\n    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py\", line 348, in _MaybeCompile\r\n    return grad_fn()  # Exit early\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py\", line 542, in <lambda>\r\n    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/ops/math_grad.py\", line 689, in _AddGrad\r\n    rx, ry = gen_array_ops._broadcast_gradient_args(sx, sy)\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py\", line 395, in _broadcast_gradient_args\r\n    name=name)\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py\", line 767, in apply_op\r\n    op_def=op_def)\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py\", line 2630, in create_op\r\n    original_op=self._default_original_op, op_def=op_def)\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py\", line 1204, in __init__\r\n    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access\r\n\r\n...which was originally created as op 'model_0/main/logits2/exp_mask', defined at:\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/cli.py\", line 113, in <module>\r\n    tf.app.run()\r\n[elided 2 identical lines from previous traceback]\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/main.py\", line 25, in main\r\n    _train(config)\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/main.py\", line 84, in _train\r\n    models = get_multi_gpu_models_new(config)\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/model_fusion.py\", line 23, in get_multi_gpu_models_new\r\n    model = Model(config, scope, rep=gpu_idx == 0)\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/model_fusion.py\", line 75, in __init__\r\n    self._build_forward()\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/basic/model_fusion.py\", line 223, in _build_forward\r\n    is_train=self.is_train, func=config.answer_func, scope='logits2')\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/my/tensorflow/nn.py\", line 107, in get_logits\r\n    is_train=is_train)\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/my/tensorflow/nn.py\", line 83, in linear_logits\r\n    logits = exp_mask(logits, mask)\r\n  File \"/nfsdata/guotong_data/bi-att-flow-dev/my/tensorflow/general.py\", line 138, in exp_mask\r\n    return tf.add(val, (1 - tf.cast(mask, 'float')) * VERY_NEGATIVE_NUMBER, name=name)\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py\", line 82, in add\r\n    result = _op_def_lib.apply_op(\"Add\", x=x, y=y, name=name)\r\n  File \"/nfsdata/guotong_data/Py35/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py\", line 767, in apply_op\r\n    op_def=op_def)\r\n\r\nInvalidArgumentError (see above for traceback): Incompatible shapes: [32,1,400] vs. [32,1,213]\r\n\t [[Node: grads_0/gradients/model_0/main/logits2/exp_mask_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape, grads_0/gradients/model_0/main/logits2/exp_mask_grad/Shape_1)]]\r\n\t [[Node: grads_0/gradients/AddN_2519/_461 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/cpu:0\", send_device=\"/job:localhost/replica:0/task:0/gpu:0\", send_device_incarnation=1, tensor_name=\"edge_289171_grads_0/gradients/AddN_2519\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/cpu:0\"]()]]\r\n\r\n\r\nProcess finished with exit code 1\r\n```\r\nthe 213 of `[32,1,213]` in stacktrace may change to 160 or other number as I change the GPU type (M40-11G , P100-15G in nvidia-docker)\r\n\r\nIn local machine it could run without the stacktrace above.\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/337781098","html_url":"https://github.com/tensorflow/tensorflow/issues/13778#issuecomment-337781098","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/13778","id":337781098,"user":{"login":"guotong1988","id":4702353,"avatar_url":"https://avatars3.githubusercontent.com/u/4702353?v=4","gravatar_id":"","url":"https://api.github.com/users/guotong1988","html_url":"https://github.com/guotong1988","followers_url":"https://api.github.com/users/guotong1988/followers","following_url":"https://api.github.com/users/guotong1988/following{/other_user}","gists_url":"https://api.github.com/users/guotong1988/gists{/gist_id}","starred_url":"https://api.github.com/users/guotong1988/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guotong1988/subscriptions","organizations_url":"https://api.github.com/users/guotong1988/orgs","repos_url":"https://api.github.com/users/guotong1988/repos","events_url":"https://api.github.com/users/guotong1988/events{/privacy}","received_events_url":"https://api.github.com/users/guotong1988/received_events","type":"User","site_admin":false},"created_at":"2017-10-19T02:24:28Z","updated_at":"2017-10-19T02:24:28Z","author_association":"CONTRIBUTOR","body":"Sorry, It is my fault.\r\nSomething wrong with my code.\r\nI haven't copy the whole folder from local machine to docker image.\r\nI only copy one edited python file.\r\nAlthough I don't know the exact reason.\r\nBut it is not the problem with the local machine and docker.\r\nIt does **could run** on docker if I copy the whole folder from local machine to docker image.\r\n\r\n"}}
{"actor":{"display_login":"guotong1988"}}