{"action":"created","issue":{"url":"https://api.github.com/repos/apache-spark-on-k8s/spark/issues/38","repository_url":"https://api.github.com/repos/apache-spark-on-k8s/spark","labels_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/issues/38/labels{/name}","comments_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/issues/38/comments","events_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/issues/38/events","html_url":"https://github.com/apache-spark-on-k8s/spark/issues/38","id":202649660,"number":38,"title":"Support custom YAML for the driver pod spec","user":{"login":"mccheah","id":8975385,"avatar_url":"https://avatars1.githubusercontent.com/u/8975385?v=4","gravatar_id":"","url":"https://api.github.com/users/mccheah","html_url":"https://github.com/mccheah","followers_url":"https://api.github.com/users/mccheah/followers","following_url":"https://api.github.com/users/mccheah/following{/other_user}","gists_url":"https://api.github.com/users/mccheah/gists{/gist_id}","starred_url":"https://api.github.com/users/mccheah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mccheah/subscriptions","organizations_url":"https://api.github.com/users/mccheah/orgs","repos_url":"https://api.github.com/users/mccheah/repos","events_url":"https://api.github.com/users/mccheah/events{/privacy}","received_events_url":"https://api.github.com/users/mccheah/received_events","type":"User","site_admin":false},"labels":[{"id":539557993,"url":"https://api.github.com/repos/apache-spark-on-k8s/spark/labels/in-beta","name":"in-beta","color":"0e8a16","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2017-01-23T21:01:54Z","updated_at":"2017-07-27T23:35:15Z","closed_at":null,"body":"We started with our base implementation, which hard-codes the pod spec to have specific fields. We then thought of the idea of supporting custom labels. However, there's plenty of things on an arbitrary user's application that could be useful to customize, such as ports and mounted volumes.\r\n\r\nThis issue therefore proposes that we support users providing arbitrary YAML files that describe the pod, or at least any modifications and augmentations they would like to make. We need to be careful when considering the API and expectations here. One mode of operation could be, the user specifies a custom file for the pod via `—driver-pod-spec-file` or an equivalent SparkConf. We can then take the user's pod spec and augment it with whatever is missing - for example, adding the Spark UI and REST submission server ports which are required for Spark.\r\n\r\nThe tricky part is the fact that the user has to specify that the container is running the Spark driver submission server, and that this container is the one that needs the custom ports open. Thus we should probably support adding `—driver-container` that must be set if the custom pod spec is set, so that we know which container we need to adjust to add the missing ports, etc. I don't know if there is any way we can make this easy to use, but in a sense usability seems to be a secondary issue here - I anticipate this will primarily be used for specific off-roading \"power-user\" scenarios."},"comment":{"url":"https://api.github.com/repos/apache-spark-on-k8s/spark/issues/comments/318515485","html_url":"https://github.com/apache-spark-on-k8s/spark/issues/38#issuecomment-318515485","issue_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/issues/38","id":318515485,"user":{"login":"erikerlandson","id":259898,"avatar_url":"https://avatars0.githubusercontent.com/u/259898?v=4","gravatar_id":"","url":"https://api.github.com/users/erikerlandson","html_url":"https://github.com/erikerlandson","followers_url":"https://api.github.com/users/erikerlandson/followers","following_url":"https://api.github.com/users/erikerlandson/following{/other_user}","gists_url":"https://api.github.com/users/erikerlandson/gists{/gist_id}","starred_url":"https://api.github.com/users/erikerlandson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erikerlandson/subscriptions","organizations_url":"https://api.github.com/users/erikerlandson/orgs","repos_url":"https://api.github.com/users/erikerlandson/repos","events_url":"https://api.github.com/users/erikerlandson/events{/privacy}","received_events_url":"https://api.github.com/users/erikerlandson/received_events","type":"User","site_admin":false},"created_at":"2017-07-27T23:35:15Z","updated_at":"2017-07-27T23:35:15Z","body":"Doing it with YAML (as opposed to PodPresets, which of course are also yaml) feels like it will be an unguarded chainsaw kind of tool.  Trying to make it both safe and easy to explain would be a hard needle to thread.  From the POV that it's for power users anyway, that isn't necessarily a deal breaker, but at the very least it would have to be documented as a dangerous-power-tool category of feature.\r\n\r\nMy inclination is to wait for pod presets, although others might be feeling varying levels of urgency."}}
{"actor":{"display_login":"erikerlandson"}}