{"push_id":2127598868,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"1e82335413bc2384073ead0d6d581c862036d0f5","before":"2014e7a789d36e376ca62b1e24636d79c1b19745","commits":[{"sha":"1e82335413bc2384073ead0d6d581c862036d0f5","author":{"name":"ArtRand","email":"2dd47f7b845c88bb4642449f698c15f235ace939@soe.ucsc.edu"},"message":"[SPARK-21842][MESOS] Support Kerberos ticket renewal and creation in Mesos\n\n## What changes were proposed in this pull request?\ntl;dr: Add a class, `MesosHadoopDelegationTokenManager` that updates delegation tokens on a schedule on the behalf of Spark Drivers. Broadcast renewed credentials to the executors.\n\n## The problem\nWe recently added Kerberos support to Mesos-based Spark jobs as well as Secrets support to the Mesos Dispatcher (SPARK-16742, SPARK-20812, respectively). However the delegation tokens have a defined expiration. This poses a problem for long running Spark jobs (e.g. Spark Streaming applications). YARN has a solution for this where a thread is scheduled to renew the tokens they reach 75% of their way to expiration. It then writes the tokens to HDFS for the executors to find (uses a monotonically increasing suffix).\n\n## This solution\nWe replace the current method in `CoarseGrainedSchedulerBackend` which used to discard the token renewal time with a protected method `fetchHadoopDelegationTokens`. Now the individual cluster backends are responsible for overriding this method to fetch and manage token renewal. The delegation tokens themselves, are still part of the `CoarseGrainedSchedulerBackend` as before.\nIn the case of Mesos renewed Credentials are broadcasted to the executors. This maintains all transfer of Credentials within Spark (as opposed to Spark-to-HDFS). It also does not require any writing of Credentials to disk. It also does not require any GC of old files.\n\n## How was this patch tested?\nManually against a Kerberized HDFS cluster.\n\nThank you for the reviews.\n\nAuthor: ArtRand <arand@soe.ucsc.edu>\n\nCloses #19272 from ArtRand/spark-21842-450-kerberos-ticket-renewal.","distinct":true,"url":"https://api.github.com/repos/apache/spark/commits/1e82335413bc2384073ead0d6d581c862036d0f5"}]}
{"actor":{"display_login":"asfgit"}}