{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16788","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16788/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16788/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16788/events","html_url":"https://github.com/apache/spark/pull/16788","id":205170073,"number":16788,"title":"[SPARK-16742] Kerberos impersonation support","user":{"login":"arinconstrio","id":10814509,"avatar_url":"https://avatars3.githubusercontent.com/u/10814509?v=3","gravatar_id":"","url":"https://api.github.com/users/arinconstrio","html_url":"https://github.com/arinconstrio","followers_url":"https://api.github.com/users/arinconstrio/followers","following_url":"https://api.github.com/users/arinconstrio/following{/other_user}","gists_url":"https://api.github.com/users/arinconstrio/gists{/gist_id}","starred_url":"https://api.github.com/users/arinconstrio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arinconstrio/subscriptions","organizations_url":"https://api.github.com/users/arinconstrio/orgs","repos_url":"https://api.github.com/users/arinconstrio/repos","events_url":"https://api.github.com/users/arinconstrio/events{/privacy}","received_events_url":"https://api.github.com/users/arinconstrio/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-02-03T14:21:52Z","updated_at":"2017-03-13T23:20:28Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16788","html_url":"https://github.com/apache/spark/pull/16788","diff_url":"https://github.com/apache/spark/pull/16788.diff","patch_url":"https://github.com/apache/spark/pull/16788.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nSolution for use kerberized hdfs and use realtime impersonation.\r\n\r\nDocument at jira [https://issues.apache.org/jira/browse/SPARK-16742](https://issues.apache.org/jira/browse/SPARK-16742)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/286274065","html_url":"https://github.com/apache/spark/pull/16788#issuecomment-286274065","issue_url":"https://api.github.com/repos/apache/spark/issues/16788","id":286274065,"user":{"login":"vanzin","id":1694083,"avatar_url":"https://avatars3.githubusercontent.com/u/1694083?v=3","gravatar_id":"","url":"https://api.github.com/users/vanzin","html_url":"https://github.com/vanzin","followers_url":"https://api.github.com/users/vanzin/followers","following_url":"https://api.github.com/users/vanzin/following{/other_user}","gists_url":"https://api.github.com/users/vanzin/gists{/gist_id}","starred_url":"https://api.github.com/users/vanzin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanzin/subscriptions","organizations_url":"https://api.github.com/users/vanzin/orgs","repos_url":"https://api.github.com/users/vanzin/repos","events_url":"https://api.github.com/users/vanzin/events{/privacy}","received_events_url":"https://api.github.com/users/vanzin/received_events","type":"User","site_admin":false},"created_at":"2017-03-13T23:20:28Z","updated_at":"2017-03-13T23:20:28Z","body":"I took a very quick look at this; Mridul and Saisai have already raised good questions about this.\r\n\r\nBut I'm a little worried that this is creating yet another way of dealing with Kerberos that will lead to two different code paths that don't share much with each other.\r\n\r\nSpark-on-YARN's support for Kerberos is really nothing complicated. The only thing that Spark needs from YARN is a method of distributing the principal's credentials. YARN just happens to have an API for that (see `setupSecurityToken` in Client.scala).\r\n\r\nOn the other side, YARN just writes the credentials to a file and sets `HADOOP_TOKEN_FILE_LOCATION` in the container process's environment to point to the file; `UserGroupInformation` reads that automatically, and things work.\r\n\r\nSo it seems to me that all that would be needed from Mesos is a way to replicate that behavior (i.e. distribute these credentials securely). The rest (including the code to deal with delegation tokens) could just be factored out of the YARN module, which should be easy now that Spark depends on Hadoop 2.6.\r\n\r\nAlso, note that `--principal` and `--keytab` are *not* required to run a Spark application in a kerberized cluster. Things should also work when you just login using `kinit` and run the Spark app. `UserGroupInformation` already handles that case for you.\r\n"}}
{"actor":{"display_login":"vanzin"}}