{"action":"created","issue":{"url":"https://api.github.com/repos/torch/nn/issues/1269","repository_url":"https://api.github.com/repos/torch/nn","labels_url":"https://api.github.com/repos/torch/nn/issues/1269/labels{/name}","comments_url":"https://api.github.com/repos/torch/nn/issues/1269/comments","events_url":"https://api.github.com/repos/torch/nn/issues/1269/events","html_url":"https://github.com/torch/nn/issues/1269","id":246395572,"number":1269,"title":"[Linear Layer] Possibly wrong gradient?","user":{"login":"CDLuminate","id":5723047,"avatar_url":"https://avatars3.githubusercontent.com/u/5723047?v=4","gravatar_id":"","url":"https://api.github.com/users/CDLuminate","html_url":"https://github.com/CDLuminate","followers_url":"https://api.github.com/users/CDLuminate/followers","following_url":"https://api.github.com/users/CDLuminate/following{/other_user}","gists_url":"https://api.github.com/users/CDLuminate/gists{/gist_id}","starred_url":"https://api.github.com/users/CDLuminate/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CDLuminate/subscriptions","organizations_url":"https://api.github.com/users/CDLuminate/orgs","repos_url":"https://api.github.com/users/CDLuminate/repos","events_url":"https://api.github.com/users/CDLuminate/events{/privacy}","received_events_url":"https://api.github.com/users/CDLuminate/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-07-28T16:22:15Z","updated_at":"2017-07-29T00:57:52Z","closed_at":null,"body":"I thought the formulations for the Linear layer should be:\r\n\r\n```\r\ncolumn y, Matrix M, column x, column b\r\n\r\ny = Wx + b\r\nδ = ∂Loss / ∂y\r\n∂Loss / ∂x = ∂y^T/∂x   ∂Loss/∂y = W^T \\cdot δ\r\n∂Loss / ∂b = ∂y^T/∂b   ∂Loss/∂y = eye \\cdot δ = δ\r\n∂Loss / ∂W = δ \\cdot x^T\r\n```\r\n\r\nSo I tried to verify those equations in pytorch:\r\n```python\r\nimport q\r\nimport torch as th\r\nfrom torch.autograd import Variable as V\r\n\r\na = V(th.rand(2,5), requires_grad=True)\r\nprint('a', a)\r\nl = th.nn.Linear(5,3)\r\ny = l.forward(a)\r\nprint('y', y)\r\n\r\ny_ = th.mm(l.weight, a.t()).t() + l.bias.expand(2,3)\r\nprint('y_', y_)\r\nprint('||y_-y||_2', (y_-y).norm())\r\n\r\ngy = th.rand(2,3)\r\nl.zero_grad()\r\ny.backward(gy)\r\nga = a.grad\r\nprint('ga', ga)\r\ngb = l.bias.grad\r\nprint('gb', gb)\r\ngw = l.weight.grad\r\nprint('gw', gw)\r\n\r\nga_ = th.mm(l.weight.data.t(), gy.t()).t()\r\nprint('ga_', ga_)\r\ngb_ = gy.t().sum(1).t()\r\nprint('gb_', gb_)\r\ngw_ = th.mm(gy.t(), a.data)\r\nprint('gw_', gw_)\r\n\r\np = lambda x,y: print((x-y).norm())\r\np(ga.data, ga_)\r\np(gw.data, gw_)\r\np(gb.data, gb_)\r\n\r\n#q.d()\r\n```\r\n\r\nThe pytorch version works, and the final three norms are all zero.\r\n\r\nHowever, I tried to verify it in lua torch and it gave me a different result:\r\n\r\n```lua\r\nrequire 'torch'\r\nrequire 'nn'\r\n\r\nx = torch.rand(1, 10)\r\nl = nn.Linear(10, 3) -- insize, outsize\r\nw = l.weight\r\nb = l.bias\r\ngy = torch.ones(1, 3)\r\n\r\nprint(w:size())\r\nprint(b:size())\r\n\r\ny = l:forward(x)\r\nl:zeroGradParameters()\r\ngx = l:backward(x, gy)\r\nl:accGradParameters(x, gy)\r\ngb = l.gradBias\r\ngw = l.gradWeight\r\n\r\nyy = x * w:t() + b\r\nggx = gy * w\r\nggb = gy:t()\r\nggw = (x:t() * gy):t()\r\n\r\nprint('yy')\r\nprint(yy)\r\nprint(y)\r\n\r\nprint('gx')\r\nprint(gx)\r\nprint(ggx)\r\n\r\nprint('gb')\r\nprint(gb)\r\nprint(ggb)\r\n\r\nprint('gw')\r\nprint(gw)\r\nprint(ggw)\r\n\r\n-- FIXME: the results differ?\r\n```\r\n\r\nThe result of the above lua torch code shows, the manually computed weight gradient `ggw`\r\nand the manually computed bias gradient `ggb` are 0.5 times as large as the original `gw` .\r\n\r\n```\r\ngb\t\r\n 2\r\n 2\r\n 2\r\n[torch.DoubleTensor of size 3]\r\n\r\n 1\r\n 1\r\n 1\r\n[torch.DoubleTensor of size 3x1]\r\n\r\ngw\t\r\n 0.7385  0.0724  0.1870  0.0149  0.7697  1.9020  0.8468  0.6302  0.1299  1.5618\r\n 0.7385  0.0724  0.1870  0.0149  0.7697  1.9020  0.8468  0.6302  0.1299  1.5618\r\n 0.7385  0.0724  0.1870  0.0149  0.7697  1.9020  0.8468  0.6302  0.1299  1.5618\r\n[torch.DoubleTensor of size 3x10]\r\n\r\n 0.3693  0.0362  0.0935  0.0075  0.3849  0.9510  0.4234  0.3151  0.0650  0.7809\r\n 0.3693  0.0362  0.0935  0.0075  0.3849  0.9510  0.4234  0.3151  0.0650  0.7809\r\n 0.3693  0.0362  0.0935  0.0075  0.3849  0.9510  0.4234  0.3151  0.0650  0.7809\r\n[torch.DoubleTensor of size 3x10]\r\n```\r\n\r\nIs this expected?"},"comment":{"url":"https://api.github.com/repos/torch/nn/issues/comments/318792283","html_url":"https://github.com/torch/nn/issues/1269#issuecomment-318792283","issue_url":"https://api.github.com/repos/torch/nn/issues/1269","id":318792283,"user":{"login":"iamalbert","id":4281586,"avatar_url":"https://avatars0.githubusercontent.com/u/4281586?v=4","gravatar_id":"","url":"https://api.github.com/users/iamalbert","html_url":"https://github.com/iamalbert","followers_url":"https://api.github.com/users/iamalbert/followers","following_url":"https://api.github.com/users/iamalbert/following{/other_user}","gists_url":"https://api.github.com/users/iamalbert/gists{/gist_id}","starred_url":"https://api.github.com/users/iamalbert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iamalbert/subscriptions","organizations_url":"https://api.github.com/users/iamalbert/orgs","repos_url":"https://api.github.com/users/iamalbert/repos","events_url":"https://api.github.com/users/iamalbert/events{/privacy}","received_events_url":"https://api.github.com/users/iamalbert/received_events","type":"User","site_admin":false},"created_at":"2017-07-29T00:57:52Z","updated_at":"2017-07-29T00:57:52Z","body":"Because you compute and accumulate gradients twice. \r\nA `backward` call is  `updateGradInput(input, gradOutput)` + `accGradParameters(input,gradOutput,scale)`\r\n"}}
{"actor":{"display_login":"iamalbert"}}