{"action":"created","issue":{"url":"https://api.github.com/repos/apache-spark-on-k8s/spark/issues/527","repository_url":"https://api.github.com/repos/apache-spark-on-k8s/spark","labels_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/issues/527/labels{/name}","comments_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/issues/527/comments","events_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/issues/527/events","html_url":"https://github.com/apache-spark-on-k8s/spark/issues/527","id":266588760,"number":527,"title":"Error: Only local python files are supported: gs://...","user":{"login":"paulreimer","id":2677,"avatar_url":"https://avatars2.githubusercontent.com/u/2677?v=4","gravatar_id":"","url":"https://api.github.com/users/paulreimer","html_url":"https://github.com/paulreimer","followers_url":"https://api.github.com/users/paulreimer/followers","following_url":"https://api.github.com/users/paulreimer/following{/other_user}","gists_url":"https://api.github.com/users/paulreimer/gists{/gist_id}","starred_url":"https://api.github.com/users/paulreimer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/paulreimer/subscriptions","organizations_url":"https://api.github.com/users/paulreimer/orgs","repos_url":"https://api.github.com/users/paulreimer/repos","events_url":"https://api.github.com/users/paulreimer/events{/privacy}","received_events_url":"https://api.github.com/users/paulreimer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-10-18T18:06:43Z","updated_at":"2017-10-18T19:48:26Z","closed_at":null,"author_association":"NONE","body":"I extended the docker image using the recent `spark-2.2.0-k8s-0.4.0-bin-2.7.3` release to add the GCS (Google Cloud Storage) connector.\r\n\r\nObserved:\r\nIt works great for scala jobs / jars with a `gs://<bucket>/` prefix - I see it creates the init container and does populate the `spark-files` from what was already in GCS. However, when I try to submit a python job (or use `--py-files`), the `spark-submit` client does not allow the `gs://` prefix and refuses the job.\r\n```\r\nError: Only local python files are supported: gs://<my_bucket_name>/pi.py\r\nRun with --help for usage help or --verbose for debug output\r\n```\r\n\r\nExpected:\r\nThe job to be allowed by `spark-submit`, the relevant files populated in an initcontainer, and available for the `spark-driver-py` and `spark-executor-py` to use successfully.\r\n\r\n(FYI To add the GCS connector, I added these lines to `spark-base` Dockerfile:)\r\n```\r\nENV hadoop_ver 2.7.4\r\n# Add Hadoop 2.x native libs\r\nADD http://www.us.apache.org/dist/hadoop/common/hadoop-${hadoop_ver}/hadoop-${hadoop_ver}.tar.gz /opt/\r\nRUN cd /opt/ && \\\r\n    tar xf hadoop-${hadoop_ver}.tar.gz && \\\r\n    ln -s hadoop-${hadoop_ver} hadoop\r\n\r\n# Add the GCS connector.\r\nADD https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar ${SPARK_HOME}/jars/\r\n```"},"comment":{"url":"https://api.github.com/repos/apache-spark-on-k8s/spark/issues/comments/337707311","html_url":"https://github.com/apache-spark-on-k8s/spark/issues/527#issuecomment-337707311","issue_url":"https://api.github.com/repos/apache-spark-on-k8s/spark/issues/527","id":337707311,"user":{"login":"paulreimer","id":2677,"avatar_url":"https://avatars2.githubusercontent.com/u/2677?v=4","gravatar_id":"","url":"https://api.github.com/users/paulreimer","html_url":"https://github.com/paulreimer","followers_url":"https://api.github.com/users/paulreimer/followers","following_url":"https://api.github.com/users/paulreimer/following{/other_user}","gists_url":"https://api.github.com/users/paulreimer/gists{/gist_id}","starred_url":"https://api.github.com/users/paulreimer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/paulreimer/subscriptions","organizations_url":"https://api.github.com/users/paulreimer/orgs","repos_url":"https://api.github.com/users/paulreimer/repos","events_url":"https://api.github.com/users/paulreimer/events{/privacy}","received_events_url":"https://api.github.com/users/paulreimer/received_events","type":"User","site_admin":false},"created_at":"2017-10-18T19:48:26Z","updated_at":"2017-10-18T19:48:26Z","author_association":"NONE","body":"Interesting, it took me so long to figure out to add `${SPARK_HOME}/conf` to the `SPARK_CLASSPATH`, to get it to pick up `core-site.xml`, but I also tried to set `fs.gs.project.id` from the command line and couldn't figure it out. Does your suggestion mean I could use `--conf spark.hadoop.fs.gs.project.id` (e.g. prefix it with `spark.hadoop`)? That would have saved me a lot of time.\r\n\r\nOne nice thing about baking it in though, is the `start-common.sh` script detects the GCE project name and writes the `fs.gs.project.id` setting in `core-site.xml` before starting. That way, at least for GCE clusters, you don't need to submit that info to `spark-submit` and can re-use the same image (it will work automatically for `gs://` URIs, then, assuming the GCE nodes have access to the storage bucket)."}}
{"actor":{"display_login":"paulreimer"}}