{"action":"created","issue":{"url":"https://api.github.com/repos/pfnet/chainer/issues/2027","repository_url":"https://api.github.com/repos/pfnet/chainer","labels_url":"https://api.github.com/repos/pfnet/chainer/issues/2027/labels{/name}","comments_url":"https://api.github.com/repos/pfnet/chainer/issues/2027/comments","events_url":"https://api.github.com/repos/pfnet/chainer/issues/2027/events","html_url":"https://github.com/pfnet/chainer/pull/2027","id":196054519,"number":2027,"title":"Out-of-core training","user":{"login":"anaruse","id":20274068,"avatar_url":"https://avatars.githubusercontent.com/u/20274068?v=3","gravatar_id":"","url":"https://api.github.com/users/anaruse","html_url":"https://github.com/anaruse","followers_url":"https://api.github.com/users/anaruse/followers","following_url":"https://api.github.com/users/anaruse/following{/other_user}","gists_url":"https://api.github.com/users/anaruse/gists{/gist_id}","starred_url":"https://api.github.com/users/anaruse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anaruse/subscriptions","organizations_url":"https://api.github.com/users/anaruse/orgs","repos_url":"https://api.github.com/users/anaruse/repos","events_url":"https://api.github.com/users/anaruse/events{/privacy}","received_events_url":"https://api.github.com/users/anaruse/received_events","type":"User","site_admin":false},"labels":[{"id":222603817,"url":"https://api.github.com/repos/pfnet/chainer/labels/feature","name":"feature","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-12-16T13:01:33Z","updated_at":"2017-02-02T11:03:16Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/pfnet/chainer/pulls/2027","html_url":"https://github.com/pfnet/chainer/pull/2027","diff_url":"https://github.com/pfnet/chainer/pull/2027.diff","patch_url":"https://github.com/pfnet/chainer/pull/2027.patch"},"body":"I'm sending a PR for out-of-core training on GPU. This enables you to perform training of very deep models such as ResNet, DenseNet and etc with suitable min-batch size on GPU.\r\n\r\nOne concern about GPU is its memory size. The amount of GPU memory is usually smaller than that of CPU memory, especially on server machines. And it is not so uncommon that you see a kind of out-of-memory error while training very large models on GPU.\r\n\r\nA basic idea behind this PR is to make use of CPU memory as a swap area. So, this PR provides a new method named _**to_swap()**_. You can use the  _**to_swap()**_ to move any variables on GPU memory to CPU memory. A method _**to_gpu()**_ is enhanced and you can use the _**to_gpu()**_ to move variables from CPU memory back to GPU memory. And this PR also provides several methods such as _**ancestors_to_swap()**_, _**ancestors_to_gpu()**_, _**interrupt_backward()**_, _**resume_backward()**_, _**enable_swapout()**_, _**disable_swapout()**_ and so on that are useful to realize appropriate data movement during forward and backward propagation.\r\n\r\nFor its usage, please refer to a file **examples/example_OOC.py**."},"comment":{"url":"https://api.github.com/repos/pfnet/chainer/issues/comments/276928400","html_url":"https://github.com/pfnet/chainer/pull/2027#issuecomment-276928400","issue_url":"https://api.github.com/repos/pfnet/chainer/issues/2027","id":276928400,"user":{"login":"anaruse","id":20274068,"avatar_url":"https://avatars.githubusercontent.com/u/20274068?v=3","gravatar_id":"","url":"https://api.github.com/users/anaruse","html_url":"https://github.com/anaruse","followers_url":"https://api.github.com/users/anaruse/followers","following_url":"https://api.github.com/users/anaruse/following{/other_user}","gists_url":"https://api.github.com/users/anaruse/gists{/gist_id}","starred_url":"https://api.github.com/users/anaruse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anaruse/subscriptions","organizations_url":"https://api.github.com/users/anaruse/orgs","repos_url":"https://api.github.com/users/anaruse/repos","events_url":"https://api.github.com/users/anaruse/events{/privacy}","received_events_url":"https://api.github.com/users/anaruse/received_events","type":"User","site_admin":false},"created_at":"2017-02-02T11:03:16Z","updated_at":"2017-02-02T11:03:16Z","body":"Added a method _**forget()**_ to the Variable class and modified back-prop methods in order to reduce memory usage. This was implemented based on the result of discussion with Chainer developers such as Okuta-san, Tokui-san, Unno-san. Thanks for the discussion!\r\n\r\nThe attached is the effect of the out-of-core training combined with the forget() method. As you can see, GPU memory usage is greatly reduced by this and very deep network like ResNet152 can be trained with large batch size. In other words, you can deal with much deeper and larger networks than ResNet152, on a single GPU.\r\n![image](https://cloud.githubusercontent.com/assets/20274068/22546316/f787ecec-e97f-11e6-9ef7-25c81d0431fa.png)\r\n\r\nFor its usage, please refer to files examples/example_OOC_(simple/VGG16/ResNet152).py."}}
{"actor":{"display_login":"anaruse"}}