{"action":"created","issue":{"url":"https://api.github.com/repos/HIPS/autograd/issues/285","repository_url":"https://api.github.com/repos/HIPS/autograd","labels_url":"https://api.github.com/repos/HIPS/autograd/issues/285/labels{/name}","comments_url":"https://api.github.com/repos/HIPS/autograd/issues/285/comments","events_url":"https://api.github.com/repos/HIPS/autograd/issues/285/events","html_url":"https://github.com/HIPS/autograd/pull/285","id":254907916,"number":285,"title":"Simplify dot grad","user":{"login":"j-towns","id":15261883,"avatar_url":"https://avatars2.githubusercontent.com/u/15261883?v=4","gravatar_id":"","url":"https://api.github.com/users/j-towns","html_url":"https://github.com/j-towns","followers_url":"https://api.github.com/users/j-towns/followers","following_url":"https://api.github.com/users/j-towns/following{/other_user}","gists_url":"https://api.github.com/users/j-towns/gists{/gist_id}","starred_url":"https://api.github.com/users/j-towns/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/j-towns/subscriptions","organizations_url":"https://api.github.com/users/j-towns/orgs","repos_url":"https://api.github.com/users/j-towns/repos","events_url":"https://api.github.com/users/j-towns/events{/privacy}","received_events_url":"https://api.github.com/users/j-towns/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2017-09-03T23:28:39Z","updated_at":"2017-09-04T19:42:13Z","closed_at":"2017-09-04T19:42:13Z","author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/HIPS/autograd/pulls/285","html_url":"https://github.com/HIPS/autograd/pull/285","diff_url":"https://github.com/HIPS/autograd/pull/285.diff","patch_url":"https://github.com/HIPS/autograd/pull/285.patch"},"body":"This started as a response to Dougal's TODO [here](https://github.com/HIPS/autograd/blob/dougal-dev/autograd/numpy/numpy_vjps.py#L297) to ensure the grad of dot didn't depend on the variable differentiated wrt.\r\n\r\nI realised that really the vjp of dot is just the adjoint of dot (as it is for all linear operators), and then I remembered the fact that the adjoint of the adjoint is just the original operator, and thought that turning the adjoint into a primitive and explicitly making its derivative equal to dot might improve performance of second and higher order derivatives.\r\n\r\nFor the n'th derivative the speedup could, I think, be linear in n (since the number of nodes was increasing by some constant amount each time a derivative was taken). Now dot and its adjoints (one for each arg) are closed under differentiation, meaning that effect won't occur. Anyway on the simple benchmarks I wrote:\r\n```\r\n    before     after       ratio\r\n[dougal-dev] [simplify-dot-grad]\r\nFirst derivatives\r\n-  239.69μs    96.71μs      0.40  bench_numpy_vjps.time_dot_0\r\n-  233.31μs   119.53μs      0.51  bench_numpy_vjps.time_dot_1\r\nSecond derivatives\r\n-  300.94μs   135.22μs      0.45  bench_numpy_vjps.time_dot_0_0\r\n-  302.74μs   134.07μs      0.44  bench_numpy_vjps.time_dot_0_1\r\n-  299.35μs   131.23μs      0.44  bench_numpy_vjps.time_dot_0_2\r\n-  522.63μs   219.09μs      0.42  bench_numpy_vjps.time_dot_1_0\r\n-  509.52μs   209.25μs      0.41  bench_numpy_vjps.time_dot_1_2\r\n-  510.89μs   208.79μs      0.41  bench_numpy_vjps.time_dot_1_1\r\n```\r\n\r\nI think there's still some scope for tidying up and maybe slightly optimizing what I've written. Also I think it would be worth doing something similar to this for the other linear primitives (some might turn out to be easier than dot).\r\n\r\nEDIT: I've just tried the benchmarks on bigger arrays and the effect of these changes seems to be reversed, i.e. everything is a bit slower on this branch than dougal-dev. Need to fix that before merging."},"comment":{"url":"https://api.github.com/repos/HIPS/autograd/issues/comments/327019887","html_url":"https://github.com/HIPS/autograd/pull/285#issuecomment-327019887","issue_url":"https://api.github.com/repos/HIPS/autograd/issues/285","id":327019887,"user":{"login":"mattjj","id":1458824,"avatar_url":"https://avatars3.githubusercontent.com/u/1458824?v=4","gravatar_id":"","url":"https://api.github.com/users/mattjj","html_url":"https://github.com/mattjj","followers_url":"https://api.github.com/users/mattjj/followers","following_url":"https://api.github.com/users/mattjj/following{/other_user}","gists_url":"https://api.github.com/users/mattjj/gists{/gist_id}","starred_url":"https://api.github.com/users/mattjj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattjj/subscriptions","organizations_url":"https://api.github.com/users/mattjj/orgs","repos_url":"https://api.github.com/users/mattjj/repos","events_url":"https://api.github.com/users/mattjj/events{/privacy}","received_events_url":"https://api.github.com/users/mattjj/received_events","type":"User","site_admin":false},"created_at":"2017-09-04T19:42:13Z","updated_at":"2017-09-04T19:42:13Z","author_association":"CONTRIBUTOR","body":"@dougalm merged these commits into the [dev-1.2](https://github.com/HIPS/autograd/tree/dev-1.2) branch, but it looks like github didn't resolve the PR. (He probably should have merged these commits into dougal-dev so that the PR would close, then switched to dev-1.2.)\r\n\r\nI'm going to close the PR because it's merged into dev-1.2 now."}}
{"actor":{"display_login":"mattjj"}}