{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/543","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/543/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/543/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/543/events","html_url":"https://github.com/tensorflow/tensorflow/issues/543","id":122873465,"number":543,"title":"Zero volatile GPU-Util but high GPU Memory Usage","user":{"login":"lglhuada","id":5636133,"avatar_url":"https://avatars2.githubusercontent.com/u/5636133?v=4","gravatar_id":"","url":"https://api.github.com/users/lglhuada","html_url":"https://github.com/lglhuada","followers_url":"https://api.github.com/users/lglhuada/followers","following_url":"https://api.github.com/users/lglhuada/following{/other_user}","gists_url":"https://api.github.com/users/lglhuada/gists{/gist_id}","starred_url":"https://api.github.com/users/lglhuada/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lglhuada/subscriptions","organizations_url":"https://api.github.com/users/lglhuada/orgs","repos_url":"https://api.github.com/users/lglhuada/repos","events_url":"https://api.github.com/users/lglhuada/events{/privacy}","received_events_url":"https://api.github.com/users/lglhuada/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"zheng-xq","id":15736910,"avatar_url":"https://avatars0.githubusercontent.com/u/15736910?v=4","gravatar_id":"","url":"https://api.github.com/users/zheng-xq","html_url":"https://github.com/zheng-xq","followers_url":"https://api.github.com/users/zheng-xq/followers","following_url":"https://api.github.com/users/zheng-xq/following{/other_user}","gists_url":"https://api.github.com/users/zheng-xq/gists{/gist_id}","starred_url":"https://api.github.com/users/zheng-xq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zheng-xq/subscriptions","organizations_url":"https://api.github.com/users/zheng-xq/orgs","repos_url":"https://api.github.com/users/zheng-xq/repos","events_url":"https://api.github.com/users/zheng-xq/events{/privacy}","received_events_url":"https://api.github.com/users/zheng-xq/received_events","type":"User","site_admin":false},"assignees":[{"login":"zheng-xq","id":15736910,"avatar_url":"https://avatars0.githubusercontent.com/u/15736910?v=4","gravatar_id":"","url":"https://api.github.com/users/zheng-xq","html_url":"https://github.com/zheng-xq","followers_url":"https://api.github.com/users/zheng-xq/followers","following_url":"https://api.github.com/users/zheng-xq/following{/other_user}","gists_url":"https://api.github.com/users/zheng-xq/gists{/gist_id}","starred_url":"https://api.github.com/users/zheng-xq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zheng-xq/subscriptions","organizations_url":"https://api.github.com/users/zheng-xq/orgs","repos_url":"https://api.github.com/users/zheng-xq/repos","events_url":"https://api.github.com/users/zheng-xq/events{/privacy}","received_events_url":"https://api.github.com/users/zheng-xq/received_events","type":"User","site_admin":false}],"milestone":null,"comments":9,"created_at":"2015-12-18T03:47:31Z","updated_at":"2017-11-06T14:27:22Z","closed_at":"2015-12-29T08:14:28Z","author_association":"NONE","body":"Hi I am running a model implemented by tensorflow with only one GPU, the GPU usage is 95% while the volatile GPU-Util is 0.\n\nSpecifically I have Tesla k40m with cuda 7.0 and cudnn 6.5v2 installed on Centos 7.0. There are three files: data_loader.py, model.py and train.py in my project. In the train.py I firstly declared \n\" with tf.device('/gpu:0'):\" and then sess.run([train_op]). When I run the code, errors raised:\n\n\"tensorflow/core/common_runtime/gpu/gpu_init.cc:45] cannot enable peer access from device ordinal 0 to device ordinal 2\"\n\nOn the other hand, I installed tensorflow with Pip.\n\nAny help are more than welcome.\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/342164198","html_url":"https://github.com/tensorflow/tensorflow/issues/543#issuecomment-342164198","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/543","id":342164198,"user":{"login":"tuobay","id":16618221,"avatar_url":"https://avatars3.githubusercontent.com/u/16618221?v=4","gravatar_id":"","url":"https://api.github.com/users/tuobay","html_url":"https://github.com/tuobay","followers_url":"https://api.github.com/users/tuobay/followers","following_url":"https://api.github.com/users/tuobay/following{/other_user}","gists_url":"https://api.github.com/users/tuobay/gists{/gist_id}","starred_url":"https://api.github.com/users/tuobay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tuobay/subscriptions","organizations_url":"https://api.github.com/users/tuobay/orgs","repos_url":"https://api.github.com/users/tuobay/repos","events_url":"https://api.github.com/users/tuobay/events{/privacy}","received_events_url":"https://api.github.com/users/tuobay/received_events","type":"User","site_admin":false},"created_at":"2017-11-06T14:27:22Z","updated_at":"2017-11-06T14:27:22Z","author_association":"NONE","body":"When I train the faster-rcnn-resnet101 model with object-detection repo, no matter how many GPUs I used, the train.py will consume all of the GPUs. e.g. when I use 1, 2 or 4 GPUs, train.py occupy all the memory of them, but only one GPU 'Volatile GPU-Util ' is 100%, the others are 0%."}}
{"actor":{"display_login":"tuobay"}}