{"action":"opened","issue":{"url":"https://api.github.com/repos/pescadores/pescador/issues/72","repository_url":"https://api.github.com/repos/pescadores/pescador","labels_url":"https://api.github.com/repos/pescadores/pescador/issues/72/labels{/name}","comments_url":"https://api.github.com/repos/pescadores/pescador/issues/72/comments","events_url":"https://api.github.com/repos/pescadores/pescador/issues/72/events","html_url":"https://github.com/pescadores/pescador/issues/72","id":218257783,"number":72,"title":"More ZMQ Streamers","user":{"login":"cjacoby","id":929267,"avatar_url":"https://avatars3.githubusercontent.com/u/929267?v=3","gravatar_id":"","url":"https://api.github.com/users/cjacoby","html_url":"https://github.com/cjacoby","followers_url":"https://api.github.com/users/cjacoby/followers","following_url":"https://api.github.com/users/cjacoby/following{/other_user}","gists_url":"https://api.github.com/users/cjacoby/gists{/gist_id}","starred_url":"https://api.github.com/users/cjacoby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cjacoby/subscriptions","organizations_url":"https://api.github.com/users/cjacoby/orgs","repos_url":"https://api.github.com/users/cjacoby/repos","events_url":"https://api.github.com/users/cjacoby/events{/privacy}","received_events_url":"https://api.github.com/users/cjacoby/received_events","type":"User","site_admin":false},"labels":[{"id":71980214,"url":"https://api.github.com/repos/pescadores/pescador/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-03-30T16:40:50Z","updated_at":"2017-03-30T16:40:50Z","closed_at":null,"body":"Just some things I've been thinking about prototyping soon but I wanted to talk through them first / get your thoughts @bmcfee. This idea isn't fully fleshed out yet.\r\n\r\nI've been doing some work with ZMQ at work lately, and I am learning that there are actually various different paradigms ZMQ is designed for. Currently, we are using only the \"paired\" mode (`request/recv`).\r\n\r\nIn particular, I am thinking about creating multiple `Streamers`, wrapped as a sort of \"Worker\" in separate python processes, streaming to one central ZMQ receiver, which supplies batches to the training process. It should be possible to have the ZMQ Workers live on other machines, as well, and therefore enabling a sort of \"CloudStreamer\" through AWS/GCP/etc. \r\n\r\nI have [this pyzmq example](http://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/devices/streamer.html) in mind. (Although it's currently unclear to me if we'd want a `Queue` device or a `Streamer` ZMQ device).\r\n\r\nA first step might be to create something like a `ZMQWorkerStreamer`, which is a `Streamer`, but sends to the intermediate Queue-like space. Then, you have a `ZMQConsumerStreamer` which might operate in much the way that `ZMQStream` does now, except with external sources.\r\n\r\n(This also might be along the lines of the asyncio version we're talked a little about, with the added bonus of external sources)."}}
{"actor":{"display_login":"cjacoby"}}