{"action":"closed","issue":{"url":"https://api.github.com/repos/treasure-data/digdag/issues/383","repository_url":"https://api.github.com/repos/treasure-data/digdag","labels_url":"https://api.github.com/repos/treasure-data/digdag/issues/383/labels{/name}","comments_url":"https://api.github.com/repos/treasure-data/digdag/issues/383/comments","events_url":"https://api.github.com/repos/treasure-data/digdag/issues/383/events","html_url":"https://github.com/treasure-data/digdag/issues/383","id":187832698,"number":383,"title":"[feature-request] Launch Amazon EMR jobs (map reduce & spark)","user":{"login":"rbparrish","id":5316371,"avatar_url":"https://avatars0.githubusercontent.com/u/5316371?v=3","gravatar_id":"","url":"https://api.github.com/users/rbparrish","html_url":"https://github.com/rbparrish","followers_url":"https://api.github.com/users/rbparrish/followers","following_url":"https://api.github.com/users/rbparrish/following{/other_user}","gists_url":"https://api.github.com/users/rbparrish/gists{/gist_id}","starred_url":"https://api.github.com/users/rbparrish/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rbparrish/subscriptions","organizations_url":"https://api.github.com/users/rbparrish/orgs","repos_url":"https://api.github.com/users/rbparrish/repos","events_url":"https://api.github.com/users/rbparrish/events{/privacy}","received_events_url":"https://api.github.com/users/rbparrish/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-11-07T21:23:32Z","updated_at":"2017-03-13T19:03:05Z","closed_at":"2017-03-13T19:03:05Z","body":"It'd be able to create & launch Amazon EMR jobs from Digdag, for running ETL or ML jobs. Either in Spark or via Hadoop/MapReduce\r\n\r\nThe AWS services of interest:\r\n- https://aws.amazon.com/emr/\r\n- https://aws.amazon.com/emr/details/spark/\r\n\r\nSome questions to discuss: \r\n(a) where do users store the program logic \r\n(b) how do users manage the dependencies required to run those programs \r\n(c) how we adequately define done \r\n(d) whether we can utilize an EMR job across multiple sequential steps, so we don't have to recreate for every task\r\n\r\nAny ideas for a simple 1st implementation? Particularly interested in using this for Spark."}}
{"actor":{"display_login":"frsyuki"}}