{"action":"created","issue":{"url":"https://api.github.com/repos/palantir/spark/issues/115","repository_url":"https://api.github.com/repos/palantir/spark","labels_url":"https://api.github.com/repos/palantir/spark/issues/115/labels{/name}","comments_url":"https://api.github.com/repos/palantir/spark/issues/115/comments","events_url":"https://api.github.com/repos/palantir/spark/issues/115/events","html_url":"https://github.com/palantir/spark/pull/115","id":212134828,"number":115,"title":"Conda Runner & full Python conda support","user":{"login":"dansanduleac","id":222827,"avatar_url":"https://avatars2.githubusercontent.com/u/222827?v=3","gravatar_id":"","url":"https://api.github.com/users/dansanduleac","html_url":"https://github.com/dansanduleac","followers_url":"https://api.github.com/users/dansanduleac/followers","following_url":"https://api.github.com/users/dansanduleac/following{/other_user}","gists_url":"https://api.github.com/users/dansanduleac/gists{/gist_id}","starred_url":"https://api.github.com/users/dansanduleac/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dansanduleac/subscriptions","organizations_url":"https://api.github.com/users/dansanduleac/orgs","repos_url":"https://api.github.com/users/dansanduleac/repos","events_url":"https://api.github.com/users/dansanduleac/events{/privacy}","received_events_url":"https://api.github.com/users/dansanduleac/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2017-03-06T14:17:06Z","updated_at":"2017-03-17T16:24:07Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/palantir/spark/pulls/115","html_url":"https://github.com/palantir/spark/pull/115","diff_url":"https://github.com/palantir/spark/pull/115.diff","patch_url":"https://github.com/palantir/spark/pull/115.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nAdd a CondaRunner abstract class that implements `main(String[])` where it sets up a conda environment when certain options are set, then delegates its arguments to an abstract method, along with the optional `CondaEnvironment`. \r\n\r\nThen, changed `PythonRunner` and `RRunner` to implement `CondaRunner` and run their respective subprocess from within the conda env, if one is set up.\r\nNOTE: actually R needs more changes, so for now `RRunner` will just fail early if conda settings are found. \r\n\r\nAs for the executor-side logic:\r\n* Updated `context.py` with a  `addCondaPackage` method, which will add the package to the locally set up conda environment (if one is defined), and for distribution on the executors\r\n* Updated `PythonRDD` to set up the desired conda env (that should match the one on the driver, which includes any additional packages supplied to `addCondaPackage`) before shelling out to python inside that env\r\n\r\n## How was this patch tested?\r\n\r\n### End-to-end python test that\r\n* bootstraps an env with only python installed (`python=3.5`)\r\n* calls `addCondaPackage('numpy=1.11.1')`\r\n* attempts to import numpy on the driver which should succeed\r\n* maps on the executor with a function that attempts to use the numpy import, and asserts that the version is the expected one\r\n\r\n### End-to-end yarn test that\r\n* runs in both client and cluster mode\r\n* essentially very similar to the above - tests that an added package is available on both driver & executor"},"comment":{"url":"https://api.github.com/repos/palantir/spark/issues/comments/287401966","html_url":"https://github.com/palantir/spark/pull/115#issuecomment-287401966","issue_url":"https://api.github.com/repos/palantir/spark/issues/115","id":287401966,"user":{"login":"ash211","id":357170,"avatar_url":"https://avatars1.githubusercontent.com/u/357170?v=3","gravatar_id":"","url":"https://api.github.com/users/ash211","html_url":"https://github.com/ash211","followers_url":"https://api.github.com/users/ash211/followers","following_url":"https://api.github.com/users/ash211/following{/other_user}","gists_url":"https://api.github.com/users/ash211/gists{/gist_id}","starred_url":"https://api.github.com/users/ash211/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ash211/subscriptions","organizations_url":"https://api.github.com/users/ash211/orgs","repos_url":"https://api.github.com/users/ash211/repos","events_url":"https://api.github.com/users/ash211/events{/privacy}","received_events_url":"https://api.github.com/users/ash211/received_events","type":"User","site_admin":false},"created_at":"2017-03-17T16:24:07Z","updated_at":"2017-03-17T16:24:07Z","body":"@justinuang does middle of next week work for you?  We're hoping to have the next release include a batch of k8s stuff that's almost finished with review: https://github.com/palantir/spark/issues/133"}}
{"actor":{"display_login":"ash211"}}