{"action":"created","issue":{"url":"https://api.github.com/repos/UrbanCCD-UChicago/plenario2/issues/52","repository_url":"https://api.github.com/repos/UrbanCCD-UChicago/plenario2","labels_url":"https://api.github.com/repos/UrbanCCD-UChicago/plenario2/issues/52/labels{/name}","comments_url":"https://api.github.com/repos/UrbanCCD-UChicago/plenario2/issues/52/comments","events_url":"https://api.github.com/repos/UrbanCCD-UChicago/plenario2/issues/52/events","html_url":"https://github.com/UrbanCCD-UChicago/plenario2/pull/52","id":278828631,"number":52,"title":"CSV ETL Pipeline","user":{"login":"HeyZoos","id":9408481,"avatar_url":"https://avatars3.githubusercontent.com/u/9408481?v=4","gravatar_id":"","url":"https://api.github.com/users/HeyZoos","html_url":"https://github.com/HeyZoos","followers_url":"https://api.github.com/users/HeyZoos/followers","following_url":"https://api.github.com/users/HeyZoos/following{/other_user}","gists_url":"https://api.github.com/users/HeyZoos/gists{/gist_id}","starred_url":"https://api.github.com/users/HeyZoos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HeyZoos/subscriptions","organizations_url":"https://api.github.com/users/HeyZoos/orgs","repos_url":"https://api.github.com/users/HeyZoos/repos","events_url":"https://api.github.com/users/HeyZoos/events{/privacy}","received_events_url":"https://api.github.com/users/HeyZoos/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-12-03T22:38:41Z","updated_at":"2017-12-03T22:57:29Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/UrbanCCD-UChicago/plenario2/pulls/52","html_url":"https://github.com/UrbanCCD-UChicago/plenario2/pull/52","diff_url":"https://github.com/UrbanCCD-UChicago/plenario2/pull/52.diff","patch_url":"https://github.com/UrbanCCD-UChicago/plenario2/pull/52.patch"},"body":"This PR adds a pipeline that can ingest and update a dataset using information provided by a `Meta` instance. (starts #16)\r\n\r\n- Adds a `download` function (starts #13)\r\n- Adds a chunked `load` function that ingests csv in 100 row chunks (closes #14)\r\n- Adds an `upsert` function\r\n- Adds a `contains` function that returns rows that would clash with an upsert to generate diffs\r\n- Adds a diffing function that generates `DataSetDiff`s with the results of upserts\r\n- Adds extra helpers to `MetaActions`\r\n- Tests it all from tip to tail with the `Mock` library\r\n\r\nI'm checking in now because the size of the changes is growing too large to effectively review. There are several issues I still need to address that I will add as extra cards. These issues being:\r\n\r\n- Pipeline currently hardcoded to work only with CSVS, still needs to ingest JSON and Shapefiles\r\n- `create_diffs` is a bit bloated and I need to break it down and refactor it\r\n- The worker supervisor is currently doing nothing, need to build an API for spinning up workers under the supervisor\r\n- Couple of small inline `TODO`s I need to address\r\n"},"comment":{"url":"https://api.github.com/repos/UrbanCCD-UChicago/plenario2/issues/comments/348825141","html_url":"https://github.com/UrbanCCD-UChicago/plenario2/pull/52#issuecomment-348825141","issue_url":"https://api.github.com/repos/UrbanCCD-UChicago/plenario2/issues/52","id":348825141,"user":{"login":"vforgione","id":4360430,"avatar_url":"https://avatars1.githubusercontent.com/u/4360430?v=4","gravatar_id":"","url":"https://api.github.com/users/vforgione","html_url":"https://github.com/vforgione","followers_url":"https://api.github.com/users/vforgione/followers","following_url":"https://api.github.com/users/vforgione/following{/other_user}","gists_url":"https://api.github.com/users/vforgione/gists{/gist_id}","starred_url":"https://api.github.com/users/vforgione/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vforgione/subscriptions","organizations_url":"https://api.github.com/users/vforgione/orgs","repos_url":"https://api.github.com/users/vforgione/repos","events_url":"https://api.github.com/users/vforgione/events{/privacy}","received_events_url":"https://api.github.com/users/vforgione/received_events","type":"User","site_admin":false},"created_at":"2017-12-03T22:57:29Z","updated_at":"2017-12-03T22:57:29Z","author_association":"OWNER","body":"general observation: i really, _really_ hate the autoformatting. it's too naive and actually makes the code less readable in a lot of areas.\r\n\r\nstuff like\r\n\r\n```\r\nRepo.one(\r\n  from(\r\n    m in Meta,\r\n    where: m.id == ^pk\r\n  )\r\n)\r\n```\r\n\r\nand \r\n\r\n```\r\nalias Plenario2.Schemas.{\r\n  DataSetConstraint,\r\n  Meta\r\n}\r\n```\r\n\r\nand adding parentheses after schema fields is just awful."}}
{"actor":{"display_login":"vforgione"}}