{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/154503085","pull_request_review_id":80681034,"id":154503085,"diff_hunk":"@@ -119,5 +139,60 @@ private[spark] object Config extends Logging {\n         \"must be a positive integer\")\n       .createWithDefault(10)\n \n+  val WAIT_FOR_APP_COMPLETION =\n+    ConfigBuilder(\"spark.kubernetes.submission.waitAppCompletion\")\n+      .doc(\"In cluster mode, whether to wait for the application to finish before exiting the \" +\n+        \"launcher process.\")\n+      .booleanConf\n+      .createWithDefault(true)\n+\n+  val REPORT_INTERVAL =\n+    ConfigBuilder(\"spark.kubernetes.report.interval\")\n+      .doc(\"Interval between reports of the current app status in cluster mode.\")\n+      .timeConf(TimeUnit.MILLISECONDS)\n+      .createWithDefaultString(\"1s\")\n+\n+  private[spark] val JARS_DOWNLOAD_LOCATION =\n+    ConfigBuilder(\"spark.kubernetes.mountDependencies.jarsDownloadDir\")\n+      .doc(\"Location to download jars to in the driver and executors. When using\" +\n+        \" spark-submit, this directory must be empty and will be mounted as an empty directory\" +\n+        \" volume on the driver and executor pod.\")\n+      .stringConf\n+      .createWithDefault(\"/var/spark-data/spark-jars\")","path":"resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala","position":117,"original_position":117,"commit_id":"0f48f036d6d68fb06d24298a11cc2c139d00735e","original_commit_id":"0f48f036d6d68fb06d24298a11cc2c139d00735e","user":{"login":"felixcheung","id":8969467,"avatar_url":"https://avatars3.githubusercontent.com/u/8969467?v=4","gravatar_id":"","url":"https://api.github.com/users/felixcheung","html_url":"https://github.com/felixcheung","followers_url":"https://api.github.com/users/felixcheung/followers","following_url":"https://api.github.com/users/felixcheung/following{/other_user}","gists_url":"https://api.github.com/users/felixcheung/gists{/gist_id}","starred_url":"https://api.github.com/users/felixcheung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/felixcheung/subscriptions","organizations_url":"https://api.github.com/users/felixcheung/orgs","repos_url":"https://api.github.com/users/felixcheung/repos","events_url":"https://api.github.com/users/felixcheung/events{/privacy}","received_events_url":"https://api.github.com/users/felixcheung/received_events","type":"User","site_admin":false},"body":"does it make sense to have a default?\r\nperhaps a comment or doc string to explain why this value as default?","created_at":"2017-12-02T18:29:43Z","updated_at":"2017-12-02T18:50:19Z","html_url":"https://github.com/apache/spark/pull/19717#discussion_r154503085","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/19717","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/154503085"},"html":{"href":"https://github.com/apache/spark/pull/19717#discussion_r154503085"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/19717"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/19717","id":151963488,"html_url":"https://github.com/apache/spark/pull/19717","diff_url":"https://github.com/apache/spark/pull/19717.diff","patch_url":"https://github.com/apache/spark/pull/19717.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/19717","number":19717,"state":"open","locked":false,"title":"[SPARK-22646] [Submission] Spark on Kubernetes - basic submission client","user":{"login":"liyinan926","id":3709137,"avatar_url":"https://avatars2.githubusercontent.com/u/3709137?v=4","gravatar_id":"","url":"https://api.github.com/users/liyinan926","html_url":"https://github.com/liyinan926","followers_url":"https://api.github.com/users/liyinan926/followers","following_url":"https://api.github.com/users/liyinan926/following{/other_user}","gists_url":"https://api.github.com/users/liyinan926/gists{/gist_id}","starred_url":"https://api.github.com/users/liyinan926/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liyinan926/subscriptions","organizations_url":"https://api.github.com/users/liyinan926/orgs","repos_url":"https://api.github.com/users/liyinan926/repos","events_url":"https://api.github.com/users/liyinan926/events{/privacy}","received_events_url":"https://api.github.com/users/liyinan926/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR contains implementation of the basic submission client for the cluster mode of Spark on Kubernetes. It's step 2 from the step-wise plan documented [here](https://github.com/apache-spark-on-k8s/spark/issues/441#issuecomment-330802935).\r\nThis addition is covered by the [SPIP](http://apache-spark-developers-list.1001551.n3.nabble.com/SPIP-Spark-on-Kubernetes-td22147.html) vote which passed on Aug 31.\r\n\r\nThis PR and #19468 together form a MVP of Spark on Kubernetes that allows users to run Spark applications that use resources locally within the driver and executor containers on Kubernetes 1.6 and up. Some changes on pom and build/test setup are copied over from #19468 to make this PR self contained and testable.\r\n\r\nThe submission client is mainly responsible for creating the Kubernetes pod that runs the Spark driver. It follows a step-based approach to construct the driver pod, as the code under the `submit.steps` package shows. The steps are orchestrated by `DriverConfigurationStepsOrchestrator`. `Client` creates the driver pod and waits for the application to complete if it's configured to do so, which is the case by default. \r\n\r\nThis PR also contains Dockerfiles of the driver and executor images. They are included because some of the environment variables set in the code would not make sense without referring to the Dockerfiles.\r\n\r\n## How was this patch tested?\r\n\r\n* The patch contains unit tests which are passing.\r\n* Manual testing: ./build/mvn -Pkubernetes clean package succeeded.\r\n* It is a subset of the entire changelist hosted at http://github.com/apache-spark-on-k8s/spark which is in active use in several organizations.\r\n* There is integration testing enabled in the fork currently hosted by PepperData which is being moved over to RiseLAB CI.\r\n* Detailed documentation on trying out the patch in its entirety is in: https://apache-spark-on-k8s.github.io/userdocs/running-on-kubernetes.html\r\n\r\ncc @rxin @felixcheung @mateiz (shepherd)\r\nk8s-big-data SIG members & contributors: @mccheah @foxish @ash211 @ssuchter @varunkatta @kimoonkim @erikerlandson @tnachen @ifilonenko @liyinan926\r\n","created_at":"2017-11-10T17:32:54Z","updated_at":"2017-12-02T18:50:19Z","closed_at":null,"merged_at":null,"merge_commit_sha":"51e48b04ff514ac06bfb69104004a10cabae2d14","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/19717/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/19717/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/19717/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/0f48f036d6d68fb06d24298a11cc2c139d00735e","head":{"label":"apache-spark-on-k8s:spark-kubernetes-4","ref":"spark-kubernetes-4","sha":"0f48f036d6d68fb06d24298a11cc2c139d00735e","user":{"login":"apache-spark-on-k8s","id":24321138,"avatar_url":"https://avatars1.githubusercontent.com/u/24321138?v=4","gravatar_id":"","url":"https://api.github.com/users/apache-spark-on-k8s","html_url":"https://github.com/apache-spark-on-k8s","followers_url":"https://api.github.com/users/apache-spark-on-k8s/followers","following_url":"https://api.github.com/users/apache-spark-on-k8s/following{/other_user}","gists_url":"https://api.github.com/users/apache-spark-on-k8s/gists{/gist_id}","starred_url":"https://api.github.com/users/apache-spark-on-k8s/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache-spark-on-k8s/subscriptions","organizations_url":"https://api.github.com/users/apache-spark-on-k8s/orgs","repos_url":"https://api.github.com/users/apache-spark-on-k8s/repos","events_url":"https://api.github.com/users/apache-spark-on-k8s/events{/privacy}","received_events_url":"https://api.github.com/users/apache-spark-on-k8s/received_events","type":"Organization","site_admin":false},"repo":{"id":75415639,"name":"spark","full_name":"apache-spark-on-k8s/spark","owner":{"login":"apache-spark-on-k8s","id":24321138,"avatar_url":"https://avatars1.githubusercontent.com/u/24321138?v=4","gravatar_id":"","url":"https://api.github.com/users/apache-spark-on-k8s","html_url":"https://github.com/apache-spark-on-k8s","followers_url":"https://api.github.com/users/apache-spark-on-k8s/followers","following_url":"https://api.github.com/users/apache-spark-on-k8s/following{/other_user}","gists_url":"https://api.github.com/users/apache-spark-on-k8s/gists{/gist_id}","starred_url":"https://api.github.com/users/apache-spark-on-k8s/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache-spark-on-k8s/subscriptions","organizations_url":"https://api.github.com/users/apache-spark-on-k8s/orgs","repos_url":"https://api.github.com/users/apache-spark-on-k8s/repos","events_url":"https://api.github.com/users/apache-spark-on-k8s/events{/privacy}","received_events_url":"https://api.github.com/users/apache-spark-on-k8s/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache-spark-on-k8s/spark","description":"Apache Spark enhanced with native Kubernetes scheduler back-end","fork":true,"url":"https://api.github.com/repos/apache-spark-on-k8s/spark","forks_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/forks","keys_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/teams","hooks_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/hooks","issue_events_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/events","assignees_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/tags","blobs_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/languages","stargazers_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/stargazers","contributors_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/contributors","subscribers_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/subscribers","subscription_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/subscription","commits_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/merges","archive_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/downloads","issues_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/deployments","created_at":"2016-12-02T17:15:19Z","updated_at":"2017-12-01T17:50:28Z","pushed_at":"2017-12-02T00:50:04Z","git_url":"git://github.com/apache-spark-on-k8s/spark.git","ssh_url":"git@github.com:apache-spark-on-k8s/spark.git","clone_url":"https://github.com/apache-spark-on-k8s/spark.git","svn_url":"https://github.com/apache-spark-on-k8s/spark","homepage":"","size":268393,"stargazers_count":301,"watchers_count":301,"language":"Scala","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":60,"mirror_url":null,"archived":false,"open_issues_count":137,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":60,"open_issues":137,"watchers":301,"default_branch":"branch-2.2-kubernetes"}},"base":{"label":"apache:master","ref":"master","sha":"999ec137a97844abbbd483dd98c7ded2f8ff356c","user":{"login":"apache","id":47359,"avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2017-12-02T15:30:42Z","pushed_at":"2017-12-02T18:45:29Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":267264,"stargazers_count":15308,"watchers_count":15308,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":14208,"mirror_url":"git://git.apache.org/spark.git","archived":false,"open_issues_count":390,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":14208,"open_issues":390,"watchers":15308,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/19717"},"html":{"href":"https://github.com/apache/spark/pull/19717"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/19717"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/19717/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/19717/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/19717/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/0f48f036d6d68fb06d24298a11cc2c139d00735e"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"felixcheung"}}