title: How to read files uploaded by spark-submit on Kubernetes
body: <p>I have Spark Jobs running on Yarn. These days I'm moving to Spark on Kubernetes.</p>
  <p>On Kubernetes I'm having an issue: files uploaded via <code>--files</code> can't be read by Spark Driver.</p>
  <p>On Yarn, as described in many answers I can read those files using <code>Source.fromFile(filename)</code>.</p>
  <p>But I can't read files in Spark on Kubernetes.</p>
  <ul>
  <li>Spark version: 3.0.1</li>
  <li>Scala version: 2.12.6</li>
  <li>deploy-mode: cluster</li>
  <li>submit commands
  <pre class="lang-none prettyprint-override"><code>$ spark-submit --class &lt;className&gt; \
        --name=&lt;jobName&gt; \
        --master=k8s://https://api-hostname:6443 \
        ...
        --deploy-mode=cluster \
        --files app.conf \
        --conf spark.kubernetes.file.upload.path=hdfs://&lt;nameservice&gt;/path/to/sparkUploads/ \
        app.jar
  </code></pre>
  </li>
  </ul>
  <p>After executing above command, <code>app.conf</code> is uploaded to <code>hdfs://&lt;nameservice&gt;/path/to/sparkUploads/spark-upload-xxxxxxx/</code>,</p>
  <p>And in Driver's pod, I found <code>app.conf</code> in <code>/tmp/spark-******/</code> directory, <code>app.jar</code> as well.</p>
  <p>But Driver can't read <code>app.conf</code>, <code>Source.fromFile(filename)</code> returns <code>null</code>, there was no permission problems.</p>
  <h3>Update 1</h3>
  <p>In Spark Web UI-&gt;&quot;Environment&quot; Tab, <code>spark://&lt;pod-name&gt;-svc.ni.svc:7078/files/app.conf</code> in &quot;Classpath Entries&quot; menu. Does this mean <code>app.conf</code> is available in classpath?</p>
  <p>On the other hand, in Spark on Yarn <code>user.dir</code> property was included in System classpath.</p>
  <hr />
  <p>I found <a href="https://issues.apache.org/jira/browse/SPARK-31726" rel="nofollow noreferrer">SPARK-31726: Make spark.files available in driver with cluster deploy mode on kubernetes</a></p>
  <h3>Update 2</h3>
  <p>I found that driver pod's <code>/opt/spark/work-dir/</code> dir was included in classpath.</p>
  <p>but <code>/opt/spark/work-dir/</code> is empty on driver pod whereas on executor pod it contains <code>app.conf</code> and <code>app.jar</code>.</p>
  <p>I think that is the problem and <a href="https://issues.apache.org/jira/browse/SPARK-31726" rel="nofollow noreferrer">SPARK-31726</a> describes this.</p>
  <h3>Update 3</h3>
  <p>After reading Jacek's answer, I tested <code>org.apache.spark.SparkFiles.getRootDirectory()</code>.</p>
  <p>It returns <code>/var/data/spark-357eb33e-1c17-4ad4-b1e8-6f878b1d8253/spark-e07d7e84-0fa7-410e-b0da-7219c412afa3/userFiles-59084588-f7f6-4ba2-a3a3-9997a780af24</code></p>
  <h3>Update 4 - work around</h3>
  <ul>
  <li>First, I make ConfigMaps to save files that I want to read driver/executors</li>
  <li>Next, The ConfigMaps are mounted on driver/executors. To mount ConfigMap, use <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template" rel="nofollow noreferrer">Pod Template</a> or <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#mounting-configmaps" rel="nofollow noreferrer">Spark Operator</a></li>
  </ul>