{"action":"created","issue":{"url":"https://api.github.com/repos/JGCRI/gcamdata/issues/401","repository_url":"https://api.github.com/repos/JGCRI/gcamdata","labels_url":"https://api.github.com/repos/JGCRI/gcamdata/issues/401/labels{/name}","comments_url":"https://api.github.com/repos/JGCRI/gcamdata/issues/401/comments","events_url":"https://api.github.com/repos/JGCRI/gcamdata/issues/401/events","html_url":"https://github.com/JGCRI/gcamdata/pull/401","id":228473901,"number":401,"title":"Various performance optimizations","user":{"login":"rplzzz","id":7209964,"avatar_url":"https://avatars3.githubusercontent.com/u/7209964?v=3","gravatar_id":"","url":"https://api.github.com/users/rplzzz","html_url":"https://github.com/rplzzz","followers_url":"https://api.github.com/users/rplzzz/followers","following_url":"https://api.github.com/users/rplzzz/following{/other_user}","gists_url":"https://api.github.com/users/rplzzz/gists{/gist_id}","starred_url":"https://api.github.com/users/rplzzz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rplzzz/subscriptions","organizations_url":"https://api.github.com/users/rplzzz/orgs","repos_url":"https://api.github.com/users/rplzzz/repos","events_url":"https://api.github.com/users/rplzzz/events{/privacy}","received_events_url":"https://api.github.com/users/rplzzz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-05-13T13:38:38Z","updated_at":"2017-05-13T14:05:19Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/JGCRI/gcamdata/pulls/401","html_url":"https://github.com/JGCRI/gcamdata/pull/401","diff_url":"https://github.com/JGCRI/gcamdata/pull/401.diff","patch_url":"https://github.com/JGCRI/gcamdata/pull/401.patch"},"body":"There is a lot more that could be done here, if anyone wants to take up the mantle.  This is more of a proof of concept than a complete treatment.\r\n\r\nSome general observations on performance:\r\n* Grouped operations are very expensive. Avoid them where possible.  _Leaving the grouping on a tbl_df will often trigger grouped operations_, so make sure to call `ungroup` when you're done with a necessary grouped operation.\r\n* Try to filter data sets as early as you can in the pipeline.  The computational complexity of everything you do scales with the size of the data.\r\n* Cross-products dramatically increase the size of the data, and therefore the complexity of everything you do to it afterward.  If you have to complete a data set, try to do so as late in the process as possible.  In particular, it is better to interpolate missing years on the final product, rather than interpolating at the beginning and carrying the interpolated data through the entire process.\r\n* Recognize common or repeated operations and do them only once, storing the results for reuse, rather than recalculating them.  \r\n* For very large tables (a few hundred thousand rows or more), there is now a `fast_left_join` function that uses a different library to do the join.  (We should probably consider a similar wrapper for large grouped operations, but I haven't done that yet.)\r\n"},"comment":{"url":"https://api.github.com/repos/JGCRI/gcamdata/issues/comments/301250488","html_url":"https://github.com/JGCRI/gcamdata/pull/401#issuecomment-301250488","issue_url":"https://api.github.com/repos/JGCRI/gcamdata/issues/401","id":301250488,"user":{"login":"bpbond","id":1956468,"avatar_url":"https://avatars1.githubusercontent.com/u/1956468?v=3","gravatar_id":"","url":"https://api.github.com/users/bpbond","html_url":"https://github.com/bpbond","followers_url":"https://api.github.com/users/bpbond/followers","following_url":"https://api.github.com/users/bpbond/following{/other_user}","gists_url":"https://api.github.com/users/bpbond/gists{/gist_id}","starred_url":"https://api.github.com/users/bpbond/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bpbond/subscriptions","organizations_url":"https://api.github.com/users/bpbond/orgs","repos_url":"https://api.github.com/users/bpbond/repos","events_url":"https://api.github.com/users/bpbond/events{/privacy}","received_events_url":"https://api.github.com/users/bpbond/received_events","type":"User","site_admin":false},"created_at":"2017-05-13T14:05:19Z","updated_at":"2017-05-13T14:05:19Z","body":"Run time for the entire driver drops from ~80 to ~70 seconds on my system. Impressive.\r\n\r\n@rplzzz note that I had to change lines 118 and 133 in `pipeline-helpers.R` to `as_tibble` to get the package to build cleanly."}}
{"actor":{"display_login":"bpbond"}}