{"action":"created","issue":{"url":"https://api.github.com/repos/twosigma/flint/issues/19","repository_url":"https://api.github.com/repos/twosigma/flint","labels_url":"https://api.github.com/repos/twosigma/flint/issues/19/labels{/name}","comments_url":"https://api.github.com/repos/twosigma/flint/issues/19/comments","events_url":"https://api.github.com/repos/twosigma/flint/issues/19/events","html_url":"https://github.com/twosigma/flint/issues/19","id":274073779,"number":19,"title":"Using flint with pyspark on yarn","user":{"login":"dadokkio","id":355183,"avatar_url":"https://avatars2.githubusercontent.com/u/355183?v=4","gravatar_id":"","url":"https://api.github.com/users/dadokkio","html_url":"https://github.com/dadokkio","followers_url":"https://api.github.com/users/dadokkio/followers","following_url":"https://api.github.com/users/dadokkio/following{/other_user}","gists_url":"https://api.github.com/users/dadokkio/gists{/gist_id}","starred_url":"https://api.github.com/users/dadokkio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dadokkio/subscriptions","organizations_url":"https://api.github.com/users/dadokkio/orgs","repos_url":"https://api.github.com/users/dadokkio/repos","events_url":"https://api.github.com/users/dadokkio/events{/privacy}","received_events_url":"https://api.github.com/users/dadokkio/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-11-15T08:34:48Z","updated_at":"2017-11-15T14:11:57Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\nI'm trying to use flint submitting a pyspark job on yarn.\r\n\r\n```\r\n>> ./bin/pyspark --master yarn --deploy-mode client --jars /opt/flint-assembly-0.2.0-SNAPSHOT.jar --py-files /opt/flint-assembly-0.2.0-SNAPSHOT.jar`\r\n[..]\r\nSparkSession available as 'spark'.\r\n>>> import ts.flint\r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 1, in <module>\r\nModuleNotFoundError: No module named 'ts'\r\n>>> import sys\r\n>>> sys.path\r\n['', '/tmp/spark-1d453f8f-379a-4f22-a7e4-cabe9dad15c5/userFiles-0b6ed883-15b1-4893-acb8-977f74b09913/flint-assembly-0.2.0-SNAPSHOT.jar', '/tmp/spark-1d453f8f-379a-4f22-a7e4-cabe9dad15c5/userFiles-0b6ed883-15b1-4893-acb8-977f74b09913', '/usr/hdp/2.6.1.0-129/spark2/python/lib/py4j-0.10.4-src.zip', '/usr/hdp/2.6.1.0-129/spark2/python', '/usr/hdp/2.6.1.0-129/spark2', '/opt/miniconda3/envs/lhd_spark/lib/python36.zip', '/opt/miniconda3/envs/lhd_spark/lib/python3.6', '/opt/miniconda3/envs/lhd_spark/lib/python3.6/lib-dynload', '/opt/miniconda3/envs/lhd_spark/lib/python3.6/site-packages', '/opt/miniconda3/envs/lhd_spark/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg']\r\n\r\n```\r\n\r\nUsing same approach on master local works properly while on yarn seems to refer to invalid path and the import fails.\r\n\r\nI was able to use the library following [this similar topic](https://stackoverflow.com/questions/36397136/importing-pyspark-packages ) extracting python code from the jar and copying it in my working directory.\r\n\r\nIs that ok or there is a better way to proceed?\r\n"},"comment":{"url":"https://api.github.com/repos/twosigma/flint/issues/comments/344603584","html_url":"https://github.com/twosigma/flint/issues/19#issuecomment-344603584","issue_url":"https://api.github.com/repos/twosigma/flint/issues/19","id":344603584,"user":{"login":"icexelloss","id":793516,"avatar_url":"https://avatars0.githubusercontent.com/u/793516?v=4","gravatar_id":"","url":"https://api.github.com/users/icexelloss","html_url":"https://github.com/icexelloss","followers_url":"https://api.github.com/users/icexelloss/followers","following_url":"https://api.github.com/users/icexelloss/following{/other_user}","gists_url":"https://api.github.com/users/icexelloss/gists{/gist_id}","starred_url":"https://api.github.com/users/icexelloss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/icexelloss/subscriptions","organizations_url":"https://api.github.com/users/icexelloss/orgs","repos_url":"https://api.github.com/users/icexelloss/repos","events_url":"https://api.github.com/users/icexelloss/events{/privacy}","received_events_url":"https://api.github.com/users/icexelloss/received_events","type":"User","site_admin":false},"created_at":"2017-11-15T14:11:57Z","updated_at":"2017-11-15T14:11:57Z","author_association":"MEMBER","body":"Hi, yeah that seems fine."}}
{"actor":{"display_login":"icexelloss"}}