{"push_id":2077988488,"size":5,"distinct_size":5,"ref":"refs/heads/fixes","head":"b3d685ee5ec426ac915200b6df49dd2a46bda56b","before":"2d0c70e6d4aa6a01ebfa1eb4810073d1a337313b","commits":[{"sha":"47927296964dba32ba9df41d897e42696e084b6e","author":{"name":"Kay Ousterhout","email":"b24382cab176d509c779a1330a414e14ac480988@kelda.io"},"message":"blueprint: Expose the per-application UI\n\nThis UI is necessary to look at the stages and tasks\nfor a particular job.","distinct":true,"url":"https://api.github.com/repos/kayousterhout/kelda-spark/commits/47927296964dba32ba9df41d897e42696e084b6e"},{"sha":"684d866cb191d830f404bd01a430a5ebc301c699","author":{"name":"Kay Ousterhout","email":"b24382cab176d509c779a1330a414e14ac480988@kelda.io"},"message":"blueprint: Set MASTER env variable on the master\n\nThe container running the Spark master needs the MASTER\nenvironment variable, so that when a new job runs on the\nmaster, the driver for that job knows how to connect to the\nSpark master (the job-specific driver runs as a separate\nprocess from the Spark master, which manages all jobs).","distinct":true,"url":"https://api.github.com/repos/kayousterhout/kelda-spark/commits/684d866cb191d830f404bd01a430a5ebc301c699"},{"sha":"0546142312e1acda7e0fcad59a805e4e56e08910","author":{"name":"Kay Ousterhout","email":"b24382cab176d509c779a1330a414e14ac480988@kelda.io"},"message":"spark: Use public IP for application UI\n\nThis commit adds functionality to fix the hostname used to link\nto the Spark driver API (from the Spark Master API) to use the\nmaster's public IP address.  This is done by setting an environment\nvariable, which is tricky because we don't know the correct value of\nthis until after the container boots (so we can't set the\nenvironment variable in the blueprint).  Right now, for the user to\nrun a job and correctly set this, the user would need to manually\nset the SPARK_PUBLIC_DNS environment variable before running a spark-submit\ncommand (or run-example or spark-shell) to submit a job.\n\nTo do this, this commit adds a spark-env.sh file that sets the environment\nvariable.  spark-env.sh is a useful (and expected) way to set environment\nvariables for Spark jobs, because the Spark job running scripts source\nenvironment variables in this file when running a job.","distinct":true,"url":"https://api.github.com/repos/kayousterhout/kelda-spark/commits/0546142312e1acda7e0fcad59a805e4e56e08910"},{"sha":"33b614a48164f8036b5242e61d2aed5daa5c383b","author":{"name":"Kay Ousterhout","email":"b24382cab176d509c779a1330a414e14ac480988@kelda.io"},"message":"spark: Remove run script\n\nThis commit changes the workers to include a spark-env.sh file.\nThe SPARK_PUBLIC_DNS enviornment variable can be set there, which\neliminates the need for the run script. This also eliminates the\nneed for the MASTER variable to be set on Spark workers.","distinct":true,"url":"https://api.github.com/repos/kayousterhout/kelda-spark/commits/33b614a48164f8036b5242e61d2aed5daa5c383b"},{"sha":"b3d685ee5ec426ac915200b6df49dd2a46bda56b","author":{"name":"Kay Ousterhout","email":"b24382cab176d509c779a1330a414e14ac480988@kelda.io"},"message":"spark: Configure necessary port access\n\nThe Spark master and workers need to talk to each other\non various ports. By default, Spark chooses those ports\nrandomly.  This commit adds a spark-defaults.conf file,\nwhich configures those ports rather than using a default\nvalue, and then enables access to those ports in\nspark.js. With this commit, running a Spark job with\na shuffle works.","distinct":true,"url":"https://api.github.com/repos/kayousterhout/kelda-spark/commits/b3d685ee5ec426ac915200b6df49dd2a46bda56b"}]}
{"actor":{"display_login":"kayousterhout"}}