title: Where does driver run and how to access SparkSession when using Spark's REST API for submission?
body: <p>Spark allow to submit jobs through REST API, <a href="https://gist.github.com/arturmkrtchyan/5d8559b2911ac951d34a#file-submit_job-sh" rel="nofollow noreferrer">example scripts</a>. There is <a href="https://github.com/ywilkof/spark-jobs-rest-client" rel="nofollow noreferrer">Spark REST Client Java API</a> which do the same, but with Java instead of curl.  </p>
  
  <p>1) Job submitted through SparkSession from my main app:</p>
  
  <pre><code>SparkConf configuration =
              new SparkConf()
                  .setAppName(name)
                  .setMaster(host)
                  .setJars(listOfJars)
                  .setAll(allPropertiesSet);
  SparkSession session =  SparkSession.builder().config(configuration).getOrCreate();
  session.sql(query).collectAsList();
  </code></pre>
  
  <hr>
  
  <p>The specific of Spark REST API is that it doesn't operate with SparkSession. So I need to package <code>MySparkJob</code> to the separate JAR and deploy to each worker.</p>
  
  <p>2) Job submitted through REST API from my main app:</p>
  
  <pre><code>SparkRestClient client = SparkRestClient
                  .builder()
                  .masterHost("cluster.master.host.address")
                  .sparkVersion("2.0.1")
                  .build();
  
          final String submissionId = client
                  .prepareJobSubmit()
                  .appName("MySparkJob!")
                  .appResource("file:/home/spark/my-spark-job.jar")
                  .mainClass("com.vbakh.jobs.MySparkJob")
                  .submit();
  </code></pre>
  
  <p><code>MySparkJob</code> code (<strong><em>where it's executed? On the same machine as REST API call? On the Spark driver? Or Spark worker?</em></strong>):</p>
  
  <pre><code>public class MySparkJob implements Serializable {
  
    public static void main(String[] args) {
       // Question 1: How to make Spark SQL Join here if I haven't got SparkSession?
       // Question 2: how to return result of the join to the driver?
    }
  }
  </code></pre>