{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/3615","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/3615/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/3615/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/3615/events","html_url":"https://github.com/pytorch/pytorch/issues/3615","id":272782028,"number":3615,"title":"pytorch distributed timeout when running with number processes > 16","user":{"login":"joe-redstone","id":33203581,"avatar_url":"https://avatars1.githubusercontent.com/u/33203581?v=4","gravatar_id":"","url":"https://api.github.com/users/joe-redstone","html_url":"https://github.com/joe-redstone","followers_url":"https://api.github.com/users/joe-redstone/followers","following_url":"https://api.github.com/users/joe-redstone/following{/other_user}","gists_url":"https://api.github.com/users/joe-redstone/gists{/gist_id}","starred_url":"https://api.github.com/users/joe-redstone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joe-redstone/subscriptions","organizations_url":"https://api.github.com/users/joe-redstone/orgs","repos_url":"https://api.github.com/users/joe-redstone/repos","events_url":"https://api.github.com/users/joe-redstone/events{/privacy}","received_events_url":"https://api.github.com/users/joe-redstone/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-11-10T00:40:36Z","updated_at":"2017-12-29T10:41:46Z","closed_at":null,"author_association":"NONE","body":"I am experimenting distributed package using a cluster with 10 computing nodes, each node has 64 cores and 256 GB RAM, no GPUs. Pytorch version is 0.2.0_3. The code runs fine when using only 16 processes or less on a single machine. But if I use 32 processes on a single machine or multiple machines (say, 4 or 6) with 16 or more processes on each machine,  it often errors out with messages like \"Read timeout\" with gloo backend or \"Connection reset by peer\" with TCP backend. After some digging into the Pytorch source code, I suspect it is likely because of the communications between processes taking too long when there are too many processes. Currently both DataChannelTCP and DataChannelGloo have a constructor that takes a \"timeout\" parameter. However the python interface does not expose this interface currently, instead it uses the constructor with default timeout set to 30s. I wonder if you can give user the option to set this. Sorry I can't post my code here but some simple code with resnet should be able to reproduce the problem."},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/354429573","html_url":"https://github.com/pytorch/pytorch/issues/3615#issuecomment-354429573","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/3615","id":354429573,"user":{"login":"apaszke","id":4583066,"avatar_url":"https://avatars3.githubusercontent.com/u/4583066?v=4","gravatar_id":"","url":"https://api.github.com/users/apaszke","html_url":"https://github.com/apaszke","followers_url":"https://api.github.com/users/apaszke/followers","following_url":"https://api.github.com/users/apaszke/following{/other_user}","gists_url":"https://api.github.com/users/apaszke/gists{/gist_id}","starred_url":"https://api.github.com/users/apaszke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apaszke/subscriptions","organizations_url":"https://api.github.com/users/apaszke/orgs","repos_url":"https://api.github.com/users/apaszke/repos","events_url":"https://api.github.com/users/apaszke/events{/privacy}","received_events_url":"https://api.github.com/users/apaszke/received_events","type":"User","site_admin":false},"created_at":"2017-12-29T10:41:46Z","updated_at":"2017-12-29T10:41:46Z","author_association":"OWNER","body":"@alsrgv Thanks a lot for the detailed writeup. That's very surprising for me, we'll need to look into the discrepancy. Changing the way you do communication shouldn't have such a dramatic effect on the training. BTW what kind of errors do you get form gloo (is it still the timeout?) and when does that happen (perhaps right before/after validation?)?"}}
{"actor":{"display_login":"apaszke"}}