title: How to solve path issue for Spark Context? AnalysisException: Path does not exist: file:/opt/workspace/
body: <p>I am running JupyterLab on my MacOS. Part of the code:</p>
  <pre><code>new_list =[]        
  for k in get_matching_s3_keys(bucket='cw-milenko-tests', prefix='Json_gzips/ticr_calculated_2', suffix='.gz'):
      new_list.append(k)
      
  
  dfs = [spark.read.json(file) for file in new_list]
  
  print (map(lambda df: len(df.schema), dfs))
  </code></pre>
  <p>I download from S3 then save to list. I got this error:</p>
  <pre><code>AnalysisException: Path does not exist: file:/opt/workspace/Json_gzips/ticr_calculated_2_2020-05-27T00-01-21.json.gz;
  </code></pre>
  <p>This is the Spark cluster I use
  <a href="https://i.stack.imgur.com/r7yBl.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/r7yBl.png" alt="enter image description here" /></a></p>
  <p>I used this repo <a href="https://github.com/andre-marcos-perez/spark-cluster-on-docker" rel="nofollow noreferrer">spark cluster on docker
  </a></p>
  <p>How to check if my Docker containers communicate?</p>
  <pre><code> docker ps
  CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS                                            NAMES
  a01477cd9316        andreper/spark-worker:latest   &quot;/bin/sh -c 'bin/spa…&quot;   4 days ago          Up 3 hours          0.0.0.0:8082-&gt;8081/tcp                           spark-worker-2
  f448de886c72        andreper/spark-worker:latest   &quot;/bin/sh -c 'bin/spa…&quot;   4 days ago          Up 3 hours          0.0.0.0:8081-&gt;8081/tcp                           spark-worker-1
  5789c47ef46e        andreper/jupyterlab:latest     &quot;/bin/sh -c 'jupyter…&quot;   4 days ago          Up 3 hours          0.0.0.0:8888-&gt;8888/tcp                           jupyterlab
  63e3d3c90ed6        andreper/spark-master:latest   &quot;/bin/sh -c 'bin/spa…&quot;   4 days ago          Up 3 hours          0.0.0.0:7077-&gt;7077/tcp, 0.0.0.0:8080-&gt;8080/tcp   spark-master
  </code></pre>
  <p>I checked the mount for jupyterlab and spark-master</p>
  <pre><code>milenko@Cloudwalkers-MacBook-Pro spark-cluster-on-docker % docker inspect -f '{{ .Mounts }}' 5789c47ef46e
  [{volume hadoop-distributed-file-system /var/lib/docker/volumes/hadoop-distributed-file-system/_data /opt/workspace local rw true }]
  milenko@Cloudwalkers-MacBook-Pro spark-cluster-on-docker % docker inspect -f '{{ .Mounts }}' 63e3d3c90ed6
  [{volume hadoop-distributed-file-system /var/lib/docker/volumes/hadoop-distributed-file-system/_data /opt/workspace local rw true }]
  </code></pre>
  <p>How to upload this file to the corresponding path in HDFS?</p>