title: Spark - pass property to spark-submit
body: <p>Spark 1.5.1 with --master yarn-cluster. What I am trying to accomplish is to pass a variable to spark-submit command that will uniquely define spawned application.  I do submit spark jobs from external application via webservice (we have another simple web layer application on dropwizard with an endpoint that submits applications). Another webservice will return status of an operation for given identifier. The flow:</p>
  
  <pre><code>SUBMIT JOB:
  MyApp -&gt; "/Dropwizard/submit-job?id=100" -&gt; Dropwizard -&gt; "spark-submit --conf=id=100" -&gt; Spark
  
  GET STATUS
  MyApp -&gt; "/Dropwizard/status?id=100" -&gt; Dropwizard -&gt; "this will get information from files that are created when application runs. Files will have id in their names"
  </code></pre>
  
  <p>Problem is <code>sparkContext.getConf().get("id");</code> returns null. 
  Can you please give me a clue how to use --conf or drop an idea how can I resolve the problem other way around?</p>