{"action":"created","issue":{"url":"https://api.github.com/repos/BVLC/caffe/issues/3580","repository_url":"https://api.github.com/repos/BVLC/caffe","labels_url":"https://api.github.com/repos/BVLC/caffe/issues/3580/labels{/name}","comments_url":"https://api.github.com/repos/BVLC/caffe/issues/3580/comments","events_url":"https://api.github.com/repos/BVLC/caffe/issues/3580/events","html_url":"https://github.com/BVLC/caffe/issues/3580","id":127828773,"number":3580,"title":"gpu problem when training imagenet","user":{"login":"jiaxinwoo","id":6374643,"avatar_url":"https://avatars.githubusercontent.com/u/6374643?v=3","gravatar_id":"","url":"https://api.github.com/users/jiaxinwoo","html_url":"https://github.com/jiaxinwoo","followers_url":"https://api.github.com/users/jiaxinwoo/followers","following_url":"https://api.github.com/users/jiaxinwoo/following{/other_user}","gists_url":"https://api.github.com/users/jiaxinwoo/gists{/gist_id}","starred_url":"https://api.github.com/users/jiaxinwoo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jiaxinwoo/subscriptions","organizations_url":"https://api.github.com/users/jiaxinwoo/orgs","repos_url":"https://api.github.com/users/jiaxinwoo/repos","events_url":"https://api.github.com/users/jiaxinwoo/events{/privacy}","received_events_url":"https://api.github.com/users/jiaxinwoo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-01-21T02:07:15Z","updated_at":"2017-02-15T14:41:55Z","closed_at":null,"body":"Ubuntu 14.04\ncuda 7.5\ncudnn v4\nk80\n\nbefore training, I check gpu info.\n$ nvidia-smi\nThu Jan 21 09:51:42 2016  \n+------------------------------------------------------+  \n| NVIDIA-SMI 352.39     Driver Version: 352.39         |  \n|-------------------------------+----------------------+----------------------+\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n|===============================+======================+======================|\n|   0  Tesla K80           On   | 0000:05:00.0     Off |                    0 |\n| N/A   55C    P8    26W / 150W |     22MiB / 11519MiB |      0%      Default |\n+-------------------------------+----------------------+----------------------+\n|   1  Tesla K80           On   | 0000:06:00.0     Off |                    0 |\n| N/A   45C    P8    69W / 150W |     22MiB / 11519MiB |      0%      Default |\n+-------------------------------+----------------------+----------------------+\n\n+-----------------------------------------------------------------------------+\n| Processes:                                                       GPU Memory |\n|  GPU       PID  Type  Process name                               Usage      |\n|=============================================================================|\n|  No running processes found                                                 |\n+-----------------------------------------------------------------------------+\n\nI successfully run MNIST example. But when I run the imagenet, \n\n$ ./build/tools/caffe train --solver=models/bvlc_reference_caffenet/solver.prototxt -gpu all\n\nit appears the following error info:\n\nI0121 09:53:55.297144  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:09.911891  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:19.214434  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:21.167176  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:23.033501  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:24.857215  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:26.761191  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:28.585139  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:30.389020  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:32.175904  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:34.028214  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:35.807531  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:37.649574  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:39.583091  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:41.497167  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:43.402312  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:45.316566  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:47.154968  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:49.037405  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:54:50.921731  2856 blocking_queue.cpp:50] Waiting for data\nI0121 09:55:24.794163  2843 solver.cpp:409]     Test net output #0: accuracy = 0.001\nI0121 09:55:24.794410  2843 solver.cpp:409]     Test net output #1: loss = 7.13962 (\\* 1 = 7.13962 loss)\nF0121 09:56:58.798702  2843 parallel.cpp:318] Check failed: error == cudaSuccess (30 vs. 0)  unknown error\n**\\* Check failure stack trace: ***\n    @     0x7f5ea63eddaa  (unknown)\n    @     0x7f5ea63edce4  (unknown)\n    @     0x7f5ea63ed6e6  (unknown)\n    @     0x7f5ea63f0687  (unknown)\n    @     0x7f5ea6abf82f  caffe::P2PSync<>::on_start()\n    @     0x7f5ea6b125af  caffe::Solver<>::Step()\n    @     0x7f5ea6b13075  caffe::Solver<>::Solve()\n    @     0x7f5ea6ac37d5  caffe::P2PSync<>::run()\n    @           0x407ac1  train()\n    @           0x4058c1  main\n    @     0x7f5ea56faec5  (unknown)\n    @           0x405fd1  (unknown)\n    @              (nil)  (unknown)\nAborted (core dumped)\n\nAfter that, I check gpu info again, it show:\n$ nvidia-smi\n\nThu Jan 21 09:58:29 2016  \n+------------------------------------------------------+  \n| NVIDIA-SMI 352.39     Driver Version: 352.39         |  \n|-------------------------------+----------------------+----------------------+\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n|===============================+======================+======================|\n|   0  ERR!               ERR!  | ERR!            ERR! |                 ERR! |\n|ERR!  ERR! ERR!    ERR! / ERR! |     22MiB / 11519MiB |    ERR!         ERR! |\n+-------------------------------+----------------------+----------------------+\n|   1  ERR!               ERR!  | ERR!            ERR! |                 ERR! |\n|ERR!  ERR! ERR!    ERR! / ERR! |     22MiB / 11519MiB |    ERR!         ERR! |\n+-------------------------------+----------------------+----------------------+\n\n+-----------------------------------------------------------------------------+\n| Processes:                                                       GPU Memory |\n|  GPU       PID  Type  Process name                               Usage      |\n|=============================================================================|\n|    0                  ERROR: GPU is lost                                    |\n|    1                  ERROR: GPU is lost                                    |\n+-----------------------------------------------------------------------------+\n\nI can't figure out, somebody has this situation before?\n"},"comment":{"url":"https://api.github.com/repos/BVLC/caffe/issues/comments/280028345","html_url":"https://github.com/BVLC/caffe/issues/3580#issuecomment-280028345","issue_url":"https://api.github.com/repos/BVLC/caffe/issues/3580","id":280028345,"user":{"login":"karaspd","id":19592664,"avatar_url":"https://avatars.githubusercontent.com/u/19592664?v=3","gravatar_id":"","url":"https://api.github.com/users/karaspd","html_url":"https://github.com/karaspd","followers_url":"https://api.github.com/users/karaspd/followers","following_url":"https://api.github.com/users/karaspd/following{/other_user}","gists_url":"https://api.github.com/users/karaspd/gists{/gist_id}","starred_url":"https://api.github.com/users/karaspd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karaspd/subscriptions","organizations_url":"https://api.github.com/users/karaspd/orgs","repos_url":"https://api.github.com/users/karaspd/repos","events_url":"https://api.github.com/users/karaspd/events{/privacy}","received_events_url":"https://api.github.com/users/karaspd/received_events","type":"User","site_admin":false},"created_at":"2017-02-15T14:41:55Z","updated_at":"2017-02-15T14:41:55Z","body":"@jiaxinwoo Have you been able to solve the problem? I got the same problem but only one of the GPU is getting lost and whole system crashes."}}
{"actor":{"display_login":"karaspd"}}