{"action":"created","issue":{"url":"https://api.github.com/repos/NVIDIA/caffe/issues/315","repository_url":"https://api.github.com/repos/NVIDIA/caffe","labels_url":"https://api.github.com/repos/NVIDIA/caffe/issues/315/labels{/name}","comments_url":"https://api.github.com/repos/NVIDIA/caffe/issues/315/comments","events_url":"https://api.github.com/repos/NVIDIA/caffe/issues/315/events","html_url":"https://github.com/NVIDIA/caffe/issues/315","id":224984811,"number":315,"title":"caffe 0.16 keeps hanging","user":{"login":"RSly","id":3428292,"avatar_url":"https://avatars1.githubusercontent.com/u/3428292?v=3","gravatar_id":"","url":"https://api.github.com/users/RSly","html_url":"https://github.com/RSly","followers_url":"https://api.github.com/users/RSly/followers","following_url":"https://api.github.com/users/RSly/following{/other_user}","gists_url":"https://api.github.com/users/RSly/gists{/gist_id}","starred_url":"https://api.github.com/users/RSly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RSly/subscriptions","organizations_url":"https://api.github.com/users/RSly/orgs","repos_url":"https://api.github.com/users/RSly/repos","events_url":"https://api.github.com/users/RSly/events{/privacy}","received_events_url":"https://api.github.com/users/RSly/received_events","type":"User","site_admin":false},"labels":[{"id":179497673,"url":"https://api.github.com/repos/NVIDIA/caffe/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":22,"created_at":"2017-04-28T05:39:32Z","updated_at":"2017-05-12T15:22:56Z","closed_at":null,"body":"Hi,\r\n\r\nI have seen this multiple times now that the training hangs after a couple of hours.\r\nI am using caffe 0.16, cudnn 6, cuda 7.5 under ubuntu and train with 3 gpus.\r\n\r\nHow can I debug this? \r\nis there a special logging mechanism for this case? there is no message, the GPU RAM is used as it should, but GPU usages are at 0\r\n\r\nalso impossible to do ctr+C to kill it, and should be killed via top/kill function\r\n\r\nhere is the last message and hours later nothing new came out\r\nI0427 21:09:09.712353  5604 sgd_solver.cpp:108] Iteration 299, lr = 0.000207384\r\n"},"comment":{"url":"https://api.github.com/repos/NVIDIA/caffe/issues/comments/301106808","html_url":"https://github.com/NVIDIA/caffe/issues/315#issuecomment-301106808","issue_url":"https://api.github.com/repos/NVIDIA/caffe/issues/315","id":301106808,"user":{"login":"drnikolaev","id":7504104,"avatar_url":"https://avatars1.githubusercontent.com/u/7504104?v=3","gravatar_id":"","url":"https://api.github.com/users/drnikolaev","html_url":"https://github.com/drnikolaev","followers_url":"https://api.github.com/users/drnikolaev/followers","following_url":"https://api.github.com/users/drnikolaev/following{/other_user}","gists_url":"https://api.github.com/users/drnikolaev/gists{/gist_id}","starred_url":"https://api.github.com/users/drnikolaev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drnikolaev/subscriptions","organizations_url":"https://api.github.com/users/drnikolaev/orgs","repos_url":"https://api.github.com/users/drnikolaev/repos","events_url":"https://api.github.com/users/drnikolaev/events{/privacy}","received_events_url":"https://api.github.com/users/drnikolaev/received_events","type":"User","site_admin":false},"created_at":"2017-05-12T15:22:56Z","updated_at":"2017-05-12T15:22:56Z","body":"@RSly hmm, is this the same https://github.com/NVIDIA/DIGITS/tree/master/examples/medical-imaging or something else?"}}
{"actor":{"display_login":"drnikolaev"}}