{"action":"created","issue":{"url":"https://api.github.com/repos/cgrand/xforms/issues/17","repository_url":"https://api.github.com/repos/cgrand/xforms","labels_url":"https://api.github.com/repos/cgrand/xforms/issues/17/labels{/name}","comments_url":"https://api.github.com/repos/cgrand/xforms/issues/17/comments","events_url":"https://api.github.com/repos/cgrand/xforms/issues/17/events","html_url":"https://github.com/cgrand/xforms/issues/17","id":260688459,"number":17,"title":"Can 'window' be used in parallel?","user":{"login":"Frozenlock","id":1152041,"avatar_url":"https://avatars1.githubusercontent.com/u/1152041?v=4","gravatar_id":"","url":"https://api.github.com/users/Frozenlock","html_url":"https://github.com/Frozenlock","followers_url":"https://api.github.com/users/Frozenlock/followers","following_url":"https://api.github.com/users/Frozenlock/following{/other_user}","gists_url":"https://api.github.com/users/Frozenlock/gists{/gist_id}","starred_url":"https://api.github.com/users/Frozenlock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Frozenlock/subscriptions","organizations_url":"https://api.github.com/users/Frozenlock/orgs","repos_url":"https://api.github.com/users/Frozenlock/repos","events_url":"https://api.github.com/users/Frozenlock/events{/privacy}","received_events_url":"https://api.github.com/users/Frozenlock/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-09-26T16:27:57Z","updated_at":"2017-09-27T10:08:57Z","closed_at":null,"author_association":"NONE","body":"I want to use `window` to compare values in a timeseries.\r\n\r\nHowever, because it uses 'surrounding' values, I'm not sure how/if it can be used in parallel.\r\nIs it possible to use it in a parallel way? (Say with core.async)"},"comment":{"url":"https://api.github.com/repos/cgrand/xforms/issues/comments/332474762","html_url":"https://github.com/cgrand/xforms/issues/17#issuecomment-332474762","issue_url":"https://api.github.com/repos/cgrand/xforms/issues/17","id":332474762,"user":{"login":"cgrand","id":47025,"avatar_url":"https://avatars0.githubusercontent.com/u/47025?v=4","gravatar_id":"","url":"https://api.github.com/users/cgrand","html_url":"https://github.com/cgrand","followers_url":"https://api.github.com/users/cgrand/followers","following_url":"https://api.github.com/users/cgrand/following{/other_user}","gists_url":"https://api.github.com/users/cgrand/gists{/gist_id}","starred_url":"https://api.github.com/users/cgrand/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cgrand/subscriptions","organizations_url":"https://api.github.com/users/cgrand/orgs","repos_url":"https://api.github.com/users/cgrand/repos","events_url":"https://api.github.com/users/cgrand/events{/privacy}","received_events_url":"https://api.github.com/users/cgrand/received_events","type":"User","site_admin":false},"created_at":"2017-09-27T10:08:57Z","updated_at":"2017-09-27T10:08:57Z","author_association":"OWNER","body":"That's an original way of using `x/window`.\r\n\r\nThe fundamental tension between transducers and parallelism is that transducers are meant to process sequential stuff (see the classic Guy Steele talk https://vimeo.com/6624203).\r\n\r\nSo when parallelizing them you need to ensure that they are stateless (or make them stateless by running them on 1-item collections) and that the returns are in the right order. Two strong constraints that make `a/pipeline` unsuitable for cheap transformations, so you have to batch inputs to make for the overhead:\r\n\r\n```clj\r\n; plain transducers\r\nuser=> (quick-bench (into [] (map inc) (range 1e5)))\r\n             Execution time mean : 4,297053 ms\r\n; throw a/pipeline in the mix and get a 160x slowdown!\r\nuser=> (quick-bench (let [in (a/chan) out (a/chan)] (a/pipeline 2 out (map inc) in) (a/onto-chan in (range 1e5)) (a/<!! (a/into [] out)) ))WARNING: Final GC required 4.450741997267441 % of runtime\r\n             Execution time mean : 687,378498 ms\r\n; 1000-item batches and a/pipeline, only 13x slower\r\nuser=> (quick-bench (let [in (a/chan 1 (x/partition 1000)) out (a/chan 1 cat)] (a/pipeline 2 out (comp cat (map inc) (x/into [])) in) (a/onto-chan in (range 1e5)) (a/<!! (a/into [] out)) ))\r\n             Execution time mean : 56,576415 ms \r\n```\r\n\r\nLooking at your code: `get-in` is relatively expensive and is called twice on each attribute (energy and time), use `*-fn` instead of `*-path` and/or move them out of `window` to a `map`.\r\n\r\nOn `partition` it can be padded (but at the other end) so that you have as many items out that you had in:\r\n```clj\r\n=> (sequence\r\n     (comp\r\n       (x/for [record %]\r\n         [record (:kwh record) (/ (get-millis (:time record)) 3.6e6)])\r\n       (x/partition 2 1 nil\r\n         (x/reduce (fn ; creative use too\r\n                     ([] nil)\r\n                     ([a]\r\n                       (if (map? a) a (assoc (first a) :power 0)))\r\n                     ([a b]\r\n                       (if-some [[ra va ta] a]\r\n                         (let [[_ vb tb] b]\r\n                           (assoc ra :power (/ (- vb va) (- tb ta))))\r\n                         b))))))\r\n     timeseries)\r\n({:time #inst \"2017-09-26T20:24:30.535-00:00\", :kwh 10, :power 12.000000002793968} {:time #inst \"2017-09-26T20:34:30.535-00:00\", :kwh 12, :power 29.999999980791472} {:time #inst \"2017-09-26T20:38:30.535-00:00\", :kwh 14, :power 15.00000000349246} {:time #inst \"2017-09-26T20:46:30.535-00:00\", :kwh 16, :power 0})\r\n```\r\n"}}
{"actor":{"display_login":"cgrand"}}