{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/4147","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/4147/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/4147/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/4147/events","html_url":"https://github.com/pytorch/pytorch/issues/4147","id":281732681,"number":4147,"title":"Parameter turns into Variable with multiple GPUs","user":{"login":"WellYoungIOE","id":19660081,"avatar_url":"https://avatars2.githubusercontent.com/u/19660081?v=4","gravatar_id":"","url":"https://api.github.com/users/WellYoungIOE","html_url":"https://github.com/WellYoungIOE","followers_url":"https://api.github.com/users/WellYoungIOE/followers","following_url":"https://api.github.com/users/WellYoungIOE/following{/other_user}","gists_url":"https://api.github.com/users/WellYoungIOE/gists{/gist_id}","starred_url":"https://api.github.com/users/WellYoungIOE/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/WellYoungIOE/subscriptions","organizations_url":"https://api.github.com/users/WellYoungIOE/orgs","repos_url":"https://api.github.com/users/WellYoungIOE/repos","events_url":"https://api.github.com/users/WellYoungIOE/events{/privacy}","received_events_url":"https://api.github.com/users/WellYoungIOE/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-12-13T12:30:35Z","updated_at":"2017-12-13T13:57:57Z","closed_at":"2017-12-13T13:57:57Z","author_association":"NONE","body":"```\r\nimport torch\r\nimport torch.nn as nn\r\nimport torch.nn.functional as func\r\nfrom torch.nn.modules import conv\r\nfrom torch.nn.modules.utils import _pair\r\nfrom torch.autograd import Variable\r\n\r\n\r\ndef _l2normalize(v, eps=1e-12):\r\n    return v / (((v**2).sum())**0.5 + eps)\r\n\r\n\r\ndef sn_norm(weights, u):\r\n\r\n    weights = weights.view(weights.shape[0], -1)\r\n    v_final = _l2normalize(torch.matmul(u, weights))\r\n    u_final = _l2normalize(torch.matmul(v_final, torch.transpose(weights, 0, 1)))\r\n\r\n    return u_final, v_final\r\n\r\n\r\nclass SNConv2D(conv._ConvNd):\r\n    def __init__(self, in_channels, out_channels, kernel_size, stride=1,\r\n                 padding=0, dilation=1, groups=1, bias=True):\r\n        kernel_size = _pair(kernel_size)\r\n        stride = _pair(stride)\r\n        padding = _pair(padding)\r\n        dilation = _pair(dilation)\r\n        super(SNConv2D, self).__init__(\r\n            in_channels, out_channels, kernel_size, stride, padding, dilation,\r\n            False, _pair(0), groups, bias)\r\n        self.mu = nn.Parameter(torch.cuda.FloatTensor(np.random.normal(0, 1, (1, out_channels))))\r\n        self.mu_new = nn.Parameter(torch.FloatTensor(np.random.normal(0, 1, (1, out_channels))))\r\n\r\n    def forward(self, input):\r\n        u_, v_ = sn_norm(self.weight.data, self.mu.data)\r\n        self.mu_new.data = u_\r\n        u_ = Variable(u_)\r\n        v_ = Variable(v_)\r\n        sigma = torch.matmul(torch.matmul(u_, self.weight.view(self.weight.shape[0], -1)), torch.transpose(v_, 0, 1))\r\n        weight_norm = self.weight / sigma\r\n        return func.conv2d(input, weight_norm, self.bias, self.stride,\r\n                           self.padding, self.dilation, self.groups)\r\n```\r\nI found `self.mu_new` turns into `Variable` in `forward()`(Maybe a copy of `self.mu_new`?) with multiple GPUs. And the code  `self.mu_new.data = u_` won't really change the data in `self.mu_new`. But it's OK with only one GPU, because `self.mu_new` is still `Parameter`(The original one?) in `forward()` with one GPU.\r\nAny way to figure it out?\r\nThanks!"},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/351398329","html_url":"https://github.com/pytorch/pytorch/issues/4147#issuecomment-351398329","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/4147","id":351398329,"user":{"login":"apaszke","id":4583066,"avatar_url":"https://avatars3.githubusercontent.com/u/4583066?v=4","gravatar_id":"","url":"https://api.github.com/users/apaszke","html_url":"https://github.com/apaszke","followers_url":"https://api.github.com/users/apaszke/followers","following_url":"https://api.github.com/users/apaszke/following{/other_user}","gists_url":"https://api.github.com/users/apaszke/gists{/gist_id}","starred_url":"https://api.github.com/users/apaszke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apaszke/subscriptions","organizations_url":"https://api.github.com/users/apaszke/orgs","repos_url":"https://api.github.com/users/apaszke/repos","events_url":"https://api.github.com/users/apaszke/events{/privacy}","received_events_url":"https://api.github.com/users/apaszke/received_events","type":"User","site_admin":false},"created_at":"2017-12-13T13:57:57Z","updated_at":"2017-12-13T13:57:57Z","author_association":"OWNER","body":"That's expected. Parameters behave exactly like Variables so it doesn't really matter."}}
{"actor":{"display_login":"apaszke"}}