{"action":"created","issue":{"url":"https://api.github.com/repos/salesforce/pytorch-qrnn/issues/6","repository_url":"https://api.github.com/repos/salesforce/pytorch-qrnn","labels_url":"https://api.github.com/repos/salesforce/pytorch-qrnn/issues/6/labels{/name}","comments_url":"https://api.github.com/repos/salesforce/pytorch-qrnn/issues/6/comments","events_url":"https://api.github.com/repos/salesforce/pytorch-qrnn/issues/6/events","html_url":"https://github.com/salesforce/pytorch-qrnn/issues/6","id":266336210,"number":6,"title":"Multi-GPU [Torch DataParallel]","user":{"login":"moscow25","id":1473764,"avatar_url":"https://avatars0.githubusercontent.com/u/1473764?v=4","gravatar_id":"","url":"https://api.github.com/users/moscow25","html_url":"https://github.com/moscow25","followers_url":"https://api.github.com/users/moscow25/followers","following_url":"https://api.github.com/users/moscow25/following{/other_user}","gists_url":"https://api.github.com/users/moscow25/gists{/gist_id}","starred_url":"https://api.github.com/users/moscow25/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/moscow25/subscriptions","organizations_url":"https://api.github.com/users/moscow25/orgs","repos_url":"https://api.github.com/users/moscow25/repos","events_url":"https://api.github.com/users/moscow25/events{/privacy}","received_events_url":"https://api.github.com/users/moscow25/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-10-18T02:00:24Z","updated_at":"2017-10-18T19:00:16Z","closed_at":null,"author_association":"NONE","body":"Could you guys get it to work with `torch.nn.DataParallel(model).cuda()`? I could not, but perhaps did not try hard enough. Can't tell if it's a wrong-GPU problem, or CuPy won't support it. \r\n\r\nRuns pretty fast on 1x GPUs though. A bit faster than 4x GPUs for vanilla LSTM, but not by much without scaling to multiple GPUs..."},"comment":{"url":"https://api.github.com/repos/salesforce/pytorch-qrnn/issues/comments/337694393","html_url":"https://github.com/salesforce/pytorch-qrnn/issues/6#issuecomment-337694393","issue_url":"https://api.github.com/repos/salesforce/pytorch-qrnn/issues/6","id":337694393,"user":{"login":"moscow25","id":1473764,"avatar_url":"https://avatars0.githubusercontent.com/u/1473764?v=4","gravatar_id":"","url":"https://api.github.com/users/moscow25","html_url":"https://github.com/moscow25","followers_url":"https://api.github.com/users/moscow25/followers","following_url":"https://api.github.com/users/moscow25/following{/other_user}","gists_url":"https://api.github.com/users/moscow25/gists{/gist_id}","starred_url":"https://api.github.com/users/moscow25/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/moscow25/subscriptions","organizations_url":"https://api.github.com/users/moscow25/orgs","repos_url":"https://api.github.com/users/moscow25/repos","events_url":"https://api.github.com/users/moscow25/events{/privacy}","received_events_url":"https://api.github.com/users/moscow25/received_events","type":"User","site_admin":false},"created_at":"2017-10-18T19:00:16Z","updated_at":"2017-10-18T19:00:16Z","author_association":"NONE","body":"Yep CuPy thing for sure. Here's my stack trace:\r\n\r\nCode is very simple. `torch.nn.Embedding` on a list of IDs, which get fed into single-layer QRNN. Works fine in CPU mode and in GPU mode with single machine and DataParallel off, just `.cuda()` the model. \r\n\r\nWill try `DistributedDataParallel` in the meantime. Thanks!\r\n\r\n```\r\n  File \"main.py\", line 510, in train\r\n    output = model(...)\r\n  File \"/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torch/nn/modules/module.py\", line 224, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py\", line 60, in forward\r\n    outputs = self.parallel_apply(replicas, inputs, kwargs)\r\n  File \"/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py\", line 70, in parallel_apply\r\n    return parallel_apply(replicas, inputs, kwargs, self.device_ids)\r\n  File \"/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torch/nn/parallel/parallel_apply.py\", line 67, in parallel_apply\r\n    raise output\r\n  File \"/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torch/nn/parallel/parallel_apply.py\", line 42, in _worker\r\n    output = module(*input, **kwargs)\r\n  File \"/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torch/nn/modules/module.py\", line 224, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/code/model_rnn.py\", line 171, in forward\r\n    output, hidden = self.rnn_jr(j_vec)\r\n  File \"/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torch/nn/modules/module.py\", line 224, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torchqrnn/qrnn.py\", line 160, in forward\r\n    input, hn = layer(input, None if hidden is None else hidden[i])\r\n  File \"/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torch/nn/modules/module.py\", line 224, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torchqrnn/qrnn.py\", line 95, in forward\r\n    C = ForgetMult()(F, Z, hidden, use_cuda=self.use_cuda)\r\n  File \"/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torch/nn/modules/module.py\", line 224, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torchqrnn/forget_mult.py\", line 172, in forward\r\n    if hidden_init is None: return GPUForgetMult()(f, x) if use_cuda else CPUForgetMult()(f, x)\r\n  File \"/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torchqrnn/forget_mult.py\", line 125, in forward\r\n    self.forget_mult(grid=grid, block=(grid_hidden_size, 1), args=[result.data_ptr(), f.data_ptr(), x.data_ptr(), seq_size, batch_size, hidden_size], stream=self.stream)\r\n  File \"cupy/cuda/function.pyx\", line 141, in cupy.cuda.function.Function.__call__\r\n  File \"cupy/cuda/function.pyx\", line 123, in cupy.cuda.function._launch\r\n  File \"cupy/cuda/driver.pyx\", line 169, in cupy.cuda.driver.launchKernel\r\n  File \"cupy/cuda/driver.pyx\", line 69, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_INVALID_HANDLE: invalid resource handle\r\nsrun: error: hsw224: task 0: Exited with exit code 1\r\n```"}}
{"actor":{"display_login":"moscow25"}}