{"action":"closed","number":4563,"pull_request":{"url":"https://api.github.com/repos/BVLC/caffe/pulls/4563","id":80542084,"html_url":"https://github.com/BVLC/caffe/pull/4563","diff_url":"https://github.com/BVLC/caffe/pull/4563.diff","patch_url":"https://github.com/BVLC/caffe/pull/4563.patch","issue_url":"https://api.github.com/repos/BVLC/caffe/issues/4563","number":4563,"state":"closed","locked":false,"title":"Merge NVIDIA's NCCL multi-GPU, switch it to python","user":{"login":"cypof","id":921950,"avatar_url":"https://avatars.githubusercontent.com/u/921950?v=3","gravatar_id":"","url":"https://api.github.com/users/cypof","html_url":"https://github.com/cypof","followers_url":"https://api.github.com/users/cypof/followers","following_url":"https://api.github.com/users/cypof/following{/other_user}","gists_url":"https://api.github.com/users/cypof/gists{/gist_id}","starred_url":"https://api.github.com/users/cypof/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cypof/subscriptions","organizations_url":"https://api.github.com/users/cypof/orgs","repos_url":"https://api.github.com/users/cypof/repos","events_url":"https://api.github.com/users/cypof/events{/privacy}","received_events_url":"https://api.github.com/users/cypof/received_events","type":"User","site_admin":false},"body":"This is an attempt to merge NVIDIA's multi-GPU work on NCCL, while fixing the open issues with python support and the parallel data pipeline. Some of them are non trivial, particularly around the GIL, so the idea is to switch to a python implementation using processes instead. NCCL supports it, and transfers are direct GPU-GPU anyway, so perf. should be the same. The code in train.py should have the same functionality as the command line and C++ version, but is much simpler and gives more flexibility to users. Custom setups from users will also be easier to share with Caffe2.\n\nTo test it, you need to clone [NCCL](https://github.com/NVIDIA/nccl), `make install` it, and set the new flag in Makefile.config.\n\nThe GIL and the fact you need to fork sub-processes before initializing CUDA made it tricky to get good performance, but it looks OK now. It should be as good as the NV branch with the new thread-pool, maybe a bit faster as the current pipeline does an in-device copy as the last step, whereas this one is fully zero-copy using CUDA IPC. I have tried on a couple of 4 and 8-GPU boxes so far and everything seems to stay busy.\n\nIf NV is interested to help testing and benchmarking, it seems layer-wise reduction and overlapping NCCL communications with compute is often slower than a single big allreduce at the end. Not sure if I'm doing something wrong, and if it can be fixed/improved. Also I don't think layer-wise works with shared weights. We need to compute actual dependencies between layers and that's better left for Caffe2. For now I simply disable layer-wise if the network has shared weights.\n\nIf we are happy with this, I suggest we deprecate multi-GPU from the command line version, and remove all associated code before 1.0. There is a lot of code we could remove around the round-robin IO, shared layers lock, root solvers etc. The complexity has grown a bit out of control, and NV had to add another layer of thread pool in their branch to make it go fast, with more Transformer functions etc. I hope we can avoid it by switching to this.\n\nTODO: breakdown the PR into simpler ones, and modularize the python code. For now it exposes a single function train() that emulates the command line. The different parts could be made easier to use and customize individually.\n","created_at":"2016-08-09T00:02:49Z","updated_at":"2017-01-17T17:12:02Z","closed_at":"2017-01-17T17:12:02Z","merged_at":"2017-01-17T17:12:02Z","merge_commit_sha":"536851c00e44d545649648b372503f52a89c0499","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/BVLC/caffe/pulls/4563/commits","review_comments_url":"https://api.github.com/repos/BVLC/caffe/pulls/4563/comments","review_comment_url":"https://api.github.com/repos/BVLC/caffe/pulls/comments{/number}","comments_url":"https://api.github.com/repos/BVLC/caffe/issues/4563/comments","statuses_url":"https://api.github.com/repos/BVLC/caffe/statuses/5f28eb1147c1abb6e5e5c7cd282218679b0d531d","head":{"label":"cypof:nccl","ref":"nccl","sha":"5f28eb1147c1abb6e5e5c7cd282218679b0d531d","user":{"login":"cypof","id":921950,"avatar_url":"https://avatars.githubusercontent.com/u/921950?v=3","gravatar_id":"","url":"https://api.github.com/users/cypof","html_url":"https://github.com/cypof","followers_url":"https://api.github.com/users/cypof/followers","following_url":"https://api.github.com/users/cypof/following{/other_user}","gists_url":"https://api.github.com/users/cypof/gists{/gist_id}","starred_url":"https://api.github.com/users/cypof/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cypof/subscriptions","organizations_url":"https://api.github.com/users/cypof/orgs","repos_url":"https://api.github.com/users/cypof/repos","events_url":"https://api.github.com/users/cypof/events{/privacy}","received_events_url":"https://api.github.com/users/cypof/received_events","type":"User","site_admin":false},"repo":{"id":25275864,"name":"caffe","full_name":"cypof/caffe","owner":{"login":"cypof","id":921950,"avatar_url":"https://avatars.githubusercontent.com/u/921950?v=3","gravatar_id":"","url":"https://api.github.com/users/cypof","html_url":"https://github.com/cypof","followers_url":"https://api.github.com/users/cypof/followers","following_url":"https://api.github.com/users/cypof/following{/other_user}","gists_url":"https://api.github.com/users/cypof/gists{/gist_id}","starred_url":"https://api.github.com/users/cypof/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cypof/subscriptions","organizations_url":"https://api.github.com/users/cypof/orgs","repos_url":"https://api.github.com/users/cypof/repos","events_url":"https://api.github.com/users/cypof/events{/privacy}","received_events_url":"https://api.github.com/users/cypof/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/cypof/caffe","description":"Caffe: a fast framework for deep learning. For the most recent version checkout the dev branch. For the latest stable release checkout the master branch.","fork":true,"url":"https://api.github.com/repos/cypof/caffe","forks_url":"https://api.github.com/repos/cypof/caffe/forks","keys_url":"https://api.github.com/repos/cypof/caffe/keys{/key_id}","collaborators_url":"https://api.github.com/repos/cypof/caffe/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/cypof/caffe/teams","hooks_url":"https://api.github.com/repos/cypof/caffe/hooks","issue_events_url":"https://api.github.com/repos/cypof/caffe/issues/events{/number}","events_url":"https://api.github.com/repos/cypof/caffe/events","assignees_url":"https://api.github.com/repos/cypof/caffe/assignees{/user}","branches_url":"https://api.github.com/repos/cypof/caffe/branches{/branch}","tags_url":"https://api.github.com/repos/cypof/caffe/tags","blobs_url":"https://api.github.com/repos/cypof/caffe/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/cypof/caffe/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/cypof/caffe/git/refs{/sha}","trees_url":"https://api.github.com/repos/cypof/caffe/git/trees{/sha}","statuses_url":"https://api.github.com/repos/cypof/caffe/statuses/{sha}","languages_url":"https://api.github.com/repos/cypof/caffe/languages","stargazers_url":"https://api.github.com/repos/cypof/caffe/stargazers","contributors_url":"https://api.github.com/repos/cypof/caffe/contributors","subscribers_url":"https://api.github.com/repos/cypof/caffe/subscribers","subscription_url":"https://api.github.com/repos/cypof/caffe/subscription","commits_url":"https://api.github.com/repos/cypof/caffe/commits{/sha}","git_commits_url":"https://api.github.com/repos/cypof/caffe/git/commits{/sha}","comments_url":"https://api.github.com/repos/cypof/caffe/comments{/number}","issue_comment_url":"https://api.github.com/repos/cypof/caffe/issues/comments{/number}","contents_url":"https://api.github.com/repos/cypof/caffe/contents/{+path}","compare_url":"https://api.github.com/repos/cypof/caffe/compare/{base}...{head}","merges_url":"https://api.github.com/repos/cypof/caffe/merges","archive_url":"https://api.github.com/repos/cypof/caffe/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/cypof/caffe/downloads","issues_url":"https://api.github.com/repos/cypof/caffe/issues{/number}","pulls_url":"https://api.github.com/repos/cypof/caffe/pulls{/number}","milestones_url":"https://api.github.com/repos/cypof/caffe/milestones{/number}","notifications_url":"https://api.github.com/repos/cypof/caffe/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/cypof/caffe/labels{/name}","releases_url":"https://api.github.com/repos/cypof/caffe/releases{/id}","deployments_url":"https://api.github.com/repos/cypof/caffe/deployments","created_at":"2014-10-15T22:12:51Z","updated_at":"2016-07-02T01:01:46Z","pushed_at":"2017-01-06T22:58:43Z","git_url":"git://github.com/cypof/caffe.git","ssh_url":"git@github.com:cypof/caffe.git","clone_url":"https://github.com/cypof/caffe.git","svn_url":"https://github.com/cypof/caffe","homepage":"http://caffe.berkeleyvision.org/","size":41869,"stargazers_count":8,"watchers_count":8,"language":"C++","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":6,"mirror_url":null,"open_issues_count":3,"forks":6,"open_issues":3,"watchers":8,"default_branch":"master"}},"base":{"label":"BVLC:master","ref":"master","sha":"f731bc4a8fc63666ae2b997e435ceb2f13752403","user":{"login":"BVLC","id":6182163,"avatar_url":"https://avatars.githubusercontent.com/u/6182163?v=3","gravatar_id":"","url":"https://api.github.com/users/BVLC","html_url":"https://github.com/BVLC","followers_url":"https://api.github.com/users/BVLC/followers","following_url":"https://api.github.com/users/BVLC/following{/other_user}","gists_url":"https://api.github.com/users/BVLC/gists{/gist_id}","starred_url":"https://api.github.com/users/BVLC/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BVLC/subscriptions","organizations_url":"https://api.github.com/users/BVLC/orgs","repos_url":"https://api.github.com/users/BVLC/repos","events_url":"https://api.github.com/users/BVLC/events{/privacy}","received_events_url":"https://api.github.com/users/BVLC/received_events","type":"Organization","site_admin":false},"repo":{"id":12791642,"name":"caffe","full_name":"BVLC/caffe","owner":{"login":"BVLC","id":6182163,"avatar_url":"https://avatars.githubusercontent.com/u/6182163?v=3","gravatar_id":"","url":"https://api.github.com/users/BVLC","html_url":"https://github.com/BVLC","followers_url":"https://api.github.com/users/BVLC/followers","following_url":"https://api.github.com/users/BVLC/following{/other_user}","gists_url":"https://api.github.com/users/BVLC/gists{/gist_id}","starred_url":"https://api.github.com/users/BVLC/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BVLC/subscriptions","organizations_url":"https://api.github.com/users/BVLC/orgs","repos_url":"https://api.github.com/users/BVLC/repos","events_url":"https://api.github.com/users/BVLC/events{/privacy}","received_events_url":"https://api.github.com/users/BVLC/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/BVLC/caffe","description":"Caffe: a fast open framework for deep learning.","fork":false,"url":"https://api.github.com/repos/BVLC/caffe","forks_url":"https://api.github.com/repos/BVLC/caffe/forks","keys_url":"https://api.github.com/repos/BVLC/caffe/keys{/key_id}","collaborators_url":"https://api.github.com/repos/BVLC/caffe/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/BVLC/caffe/teams","hooks_url":"https://api.github.com/repos/BVLC/caffe/hooks","issue_events_url":"https://api.github.com/repos/BVLC/caffe/issues/events{/number}","events_url":"https://api.github.com/repos/BVLC/caffe/events","assignees_url":"https://api.github.com/repos/BVLC/caffe/assignees{/user}","branches_url":"https://api.github.com/repos/BVLC/caffe/branches{/branch}","tags_url":"https://api.github.com/repos/BVLC/caffe/tags","blobs_url":"https://api.github.com/repos/BVLC/caffe/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/BVLC/caffe/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/BVLC/caffe/git/refs{/sha}","trees_url":"https://api.github.com/repos/BVLC/caffe/git/trees{/sha}","statuses_url":"https://api.github.com/repos/BVLC/caffe/statuses/{sha}","languages_url":"https://api.github.com/repos/BVLC/caffe/languages","stargazers_url":"https://api.github.com/repos/BVLC/caffe/stargazers","contributors_url":"https://api.github.com/repos/BVLC/caffe/contributors","subscribers_url":"https://api.github.com/repos/BVLC/caffe/subscribers","subscription_url":"https://api.github.com/repos/BVLC/caffe/subscription","commits_url":"https://api.github.com/repos/BVLC/caffe/commits{/sha}","git_commits_url":"https://api.github.com/repos/BVLC/caffe/git/commits{/sha}","comments_url":"https://api.github.com/repos/BVLC/caffe/comments{/number}","issue_comment_url":"https://api.github.com/repos/BVLC/caffe/issues/comments{/number}","contents_url":"https://api.github.com/repos/BVLC/caffe/contents/{+path}","compare_url":"https://api.github.com/repos/BVLC/caffe/compare/{base}...{head}","merges_url":"https://api.github.com/repos/BVLC/caffe/merges","archive_url":"https://api.github.com/repos/BVLC/caffe/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/BVLC/caffe/downloads","issues_url":"https://api.github.com/repos/BVLC/caffe/issues{/number}","pulls_url":"https://api.github.com/repos/BVLC/caffe/pulls{/number}","milestones_url":"https://api.github.com/repos/BVLC/caffe/milestones{/number}","notifications_url":"https://api.github.com/repos/BVLC/caffe/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/BVLC/caffe/labels{/name}","releases_url":"https://api.github.com/repos/BVLC/caffe/releases{/id}","deployments_url":"https://api.github.com/repos/BVLC/caffe/deployments","created_at":"2013-09-12T18:39:48Z","updated_at":"2017-01-17T17:04:45Z","pushed_at":"2017-01-17T17:12:02Z","git_url":"git://github.com/BVLC/caffe.git","ssh_url":"git@github.com:BVLC/caffe.git","clone_url":"https://github.com/BVLC/caffe.git","svn_url":"https://github.com/BVLC/caffe","homepage":"http://caffe.berkeleyvision.org/","size":49375,"stargazers_count":15348,"watchers_count":15348,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":9517,"mirror_url":null,"open_issues_count":968,"forks":9517,"open_issues":968,"watchers":15348,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/BVLC/caffe/pulls/4563"},"html":{"href":"https://github.com/BVLC/caffe/pull/4563"},"issue":{"href":"https://api.github.com/repos/BVLC/caffe/issues/4563"},"comments":{"href":"https://api.github.com/repos/BVLC/caffe/issues/4563/comments"},"review_comments":{"href":"https://api.github.com/repos/BVLC/caffe/pulls/4563/comments"},"review_comment":{"href":"https://api.github.com/repos/BVLC/caffe/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/BVLC/caffe/pulls/4563/commits"},"statuses":{"href":"https://api.github.com/repos/BVLC/caffe/statuses/5f28eb1147c1abb6e5e5c7cd282218679b0d531d"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"shelhamer","id":311113,"avatar_url":"https://avatars.githubusercontent.com/u/311113?v=3","gravatar_id":"","url":"https://api.github.com/users/shelhamer","html_url":"https://github.com/shelhamer","followers_url":"https://api.github.com/users/shelhamer/followers","following_url":"https://api.github.com/users/shelhamer/following{/other_user}","gists_url":"https://api.github.com/users/shelhamer/gists{/gist_id}","starred_url":"https://api.github.com/users/shelhamer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shelhamer/subscriptions","organizations_url":"https://api.github.com/users/shelhamer/orgs","repos_url":"https://api.github.com/users/shelhamer/repos","events_url":"https://api.github.com/users/shelhamer/events{/privacy}","received_events_url":"https://api.github.com/users/shelhamer/received_events","type":"User","site_admin":false},"comments":14,"review_comments":47,"maintainer_can_modify":false,"commits":5,"additions":1022,"deletions":883,"changed_files":52}}
{"actor":{"display_login":"shelhamer"}}