{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/1099","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/1099/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/1099/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/1099/events","html_url":"https://github.com/pytorch/pytorch/issues/1099","id":216947272,"number":1099,"title":"Implementation of SGD + Momentum","user":{"login":"keskarnitish","id":5945552,"avatar_url":"https://avatars0.githubusercontent.com/u/5945552?v=3","gravatar_id":"","url":"https://api.github.com/users/keskarnitish","html_url":"https://github.com/keskarnitish","followers_url":"https://api.github.com/users/keskarnitish/followers","following_url":"https://api.github.com/users/keskarnitish/following{/other_user}","gists_url":"https://api.github.com/users/keskarnitish/gists{/gist_id}","starred_url":"https://api.github.com/users/keskarnitish/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keskarnitish/subscriptions","organizations_url":"https://api.github.com/users/keskarnitish/orgs","repos_url":"https://api.github.com/users/keskarnitish/repos","events_url":"https://api.github.com/users/keskarnitish/events{/privacy}","received_events_url":"https://api.github.com/users/keskarnitish/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-03-25T03:55:51Z","updated_at":"2017-03-25T05:34:49Z","closed_at":null,"body":"I have been looking at the implementation of SGD + Momentum in PyTorch and noticed something a bit different from how other packages (and papers) describe it. For the moment, let's focus solely on (classical) momentum and not Nesterov's version.  \r\n\r\nAt the time of writing, the implementation reads:\r\n\r\n ```\r\n               if momentum != 0:\r\n                    param_state = self.state[p]\r\n                    if 'momentum_buffer' not in param_state:\r\n                        buf = param_state['momentum_buffer'] = d_p.clone()\r\n                    else:\r\n                        buf = param_state['momentum_buffer']\r\n                        buf.mul_(momentum).add_(1 - dampening, d_p)\r\n                    if nesterov:\r\n                        d_p = d_p.add(momentum, buf)\r\n                    else:\r\n                        d_p = buf\r\n\r\n                p.data.add_(-group['lr'], d_p)\r\n``` \r\n\r\nMathematically, if we denote the momentum buffer by `v` and assume that `dampening=0`, at every iteration, the buffer is updated as `v = m*v + g` and the step is `∆x = lr * v`. Notice that the learning rate `lr` hits the momentum term `v` as well as the gradient. To me, this is different from what classical momentum is, and also differs from how other packages implement SGD+M.\r\n\r\nLet us contrast this with the Sutskever et. al. paper and other commonly used pacakges such as Lasagne, Keras, Neon, etc.\r\n\r\n## [Sutskever et. al.](http://www.jmlr.org/proceedings/papers/v28/sutskever13.pdf)\r\nThe snippet of the relevant section is pasted below. \r\n![Sutskever et. al.](http://i.imgur.com/QJelodE.png)\r\n\r\nRetaining the syntax from above, the algorithm updates `v` as `v = m*v - lr * g` with the step `∆x = v`. So, the learning rate `lr` only hits the gradient. It does not (explicitly) influence the effect of the momentum term which is in contrast with PyTorch's implementation. \r\n\r\n# [Lasagne](https://github.com/Lasagne/Lasagne/blob/master/lasagne/updates.py#L217)\r\n\r\nLasagne employs the same rule as suggested in Sutskever for momentum. \r\n\r\n```\r\n    for param in params:\r\n        value = param.get_value(borrow=True)\r\n        velocity = theano.shared(np.zeros(value.shape, dtype=value.dtype),\r\n                                 broadcastable=param.broadcastable)\r\n        x = momentum * velocity + updates[param]\r\n        updates[velocity] = x - param\r\n```\r\n\r\n# [Keras](https://github.com/fchollet/keras/blob/master/keras/optimizers.py#L141)\r\n\r\nSame for Keras:\r\n```\r\n       for p, g, m in zip(params, grads, moments):\r\n            v = self.momentum * m - lr * g  # velocity\r\n            self.updates.append(K.update(m, v))\r\n\r\n            if self.nesterov:\r\n                new_p = p + self.momentum * v - lr * g\r\n            else:\r\n                new_p = p + v\r\n```\r\n\r\n# [Neon](https://github.com/NervanaSystems/neon/blob/master/neon/optimizers/optimizer.py#L520)\r\n\r\nand Neon.\r\n```\r\n                velocity[:] = self.momentum_coef * velocity - lrate * grad\r\n\r\n                # Nesterov accelerated gradient (NAG) is implemented the same\r\n                # as in torch's \"sgd.lua\". It's a reformulation of Sutskever's\r\n                # NAG equation found in \"On the importance of initialization\r\n                # and momentum in deep learning\".\r\n                if self.nesterov:\r\n                    param[:] = param + self.momentum_coef * velocity -\\\r\n                               lrate * grad\r\n                else:\r\n                    param[:] = param + velocity\r\n```\r\nIs the disparity true or am I missing something important?\r\n\r\nThe difference between the two implementations is not insignificant and especially so when `lr` is reduced along the way. If my claim is true, maybe we could update the reference (I'm not sure what that would be) or include the above version in the SGD code (I can take this up if necessary)? "},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/289190614","html_url":"https://github.com/pytorch/pytorch/issues/1099#issuecomment-289190614","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/1099","id":289190614,"user":{"login":"colesbury","id":655866,"avatar_url":"https://avatars2.githubusercontent.com/u/655866?v=3","gravatar_id":"","url":"https://api.github.com/users/colesbury","html_url":"https://github.com/colesbury","followers_url":"https://api.github.com/users/colesbury/followers","following_url":"https://api.github.com/users/colesbury/following{/other_user}","gists_url":"https://api.github.com/users/colesbury/gists{/gist_id}","starred_url":"https://api.github.com/users/colesbury/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/colesbury/subscriptions","organizations_url":"https://api.github.com/users/colesbury/orgs","repos_url":"https://api.github.com/users/colesbury/repos","events_url":"https://api.github.com/users/colesbury/events{/privacy}","received_events_url":"https://api.github.com/users/colesbury/received_events","type":"User","site_admin":false},"created_at":"2017-03-25T05:34:49Z","updated_at":"2017-03-25T05:34:49Z","body":"For a fixed learning rate, the two formulations are equivalent. The Torch formulation is chosen because the the step size is directly proportional to the learning rate. This means that if you decrease the learning rate, the step size decreases immediately, and not after some number of iterations, which is generally what you want."}}
{"actor":{"display_login":"colesbury"}}