{"action":"opened","issue":{"url":"https://api.github.com/repos/intel-analytics/BigDL/issues/1812","repository_url":"https://api.github.com/repos/intel-analytics/BigDL","labels_url":"https://api.github.com/repos/intel-analytics/BigDL/issues/1812/labels{/name}","comments_url":"https://api.github.com/repos/intel-analytics/BigDL/issues/1812/comments","events_url":"https://api.github.com/repos/intel-analytics/BigDL/issues/1812/events","html_url":"https://github.com/intel-analytics/BigDL/issues/1812","id":270208383,"number":1812,"title":"should not throw exception if there's no spark-bigdl.conf in python zip file","user":{"login":"shane-huang","id":1995599,"avatar_url":"https://avatars0.githubusercontent.com/u/1995599?v=4","gravatar_id":"","url":"https://api.github.com/users/shane-huang","html_url":"https://github.com/shane-huang","followers_url":"https://api.github.com/users/shane-huang/followers","following_url":"https://api.github.com/users/shane-huang/following{/other_user}","gists_url":"https://api.github.com/users/shane-huang/gists{/gist_id}","starred_url":"https://api.github.com/users/shane-huang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shane-huang/subscriptions","organizations_url":"https://api.github.com/users/shane-huang/orgs","repos_url":"https://api.github.com/users/shane-huang/repos","events_url":"https://api.github.com/users/shane-huang/events{/privacy}","received_events_url":"https://api.github.com/users/shane-huang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-11-01T06:26:59Z","updated_at":"2017-11-01T06:26:59Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"BigDL will look for spark-bigdl.conf in every user uploaded zips in --py-files.  But if it is a user appliction package, it may not contain a conf. A default conf should be used and no exception thrown.  \r\n\r\n```python\r\n---------------------------------------------------------------------------\r\nKeyError                                  Traceback (most recent call last)\r\n<ipython-input-2-c7f82e7fe39a> in <module>()\r\n      5 from bigdl.nn.criterion import *\r\n      6 \r\n----> 7 init_engine()\r\n      8 #conf = create_spark_conf()\r\n      9 #conf.setMaster(\"local[1]\")\r\n\r\n/tmp/spark-21b61136-a031-4bc7-8e5b-032bbd07f63c/userFiles-d6fa7616-6c6e-445d-8233-153c1278d34d/bigdl-0.3.0-SNAPSHOT-python-api.zip/bigdl/util/common.py in init_engine(bigdl_type)\r\n    367 \r\n    368 def init_engine(bigdl_type=\"float\"):\r\n--> 369     callBigDlFunc(bigdl_type, \"initEngine\")\r\n    370 \r\n    371 \r\n\r\n/tmp/spark-21b61136-a031-4bc7-8e5b-032bbd07f63c/userFiles-d6fa7616-6c6e-445d-8233-153c1278d34d/bigdl-0.3.0-SNAPSHOT-python-api.zip/bigdl/util/common.py in callBigDlFunc(bigdl_type, name, *args)\r\n    453 def callBigDlFunc(bigdl_type, name, *args):\r\n    454     \"\"\" Call API in PythonBigDL \"\"\"\r\n--> 455     jinstance = JavaCreator.instance(bigdl_type=bigdl_type).value\r\n    456     sc = get_spark_context()\r\n    457     api = getattr(jinstance, name)\r\n\r\n/tmp/spark-21b61136-a031-4bc7-8e5b-032bbd07f63c/userFiles-d6fa7616-6c6e-445d-8233-153c1278d34d/bigdl-0.3.0-SNAPSHOT-python-api.zip/bigdl/util/common.py in instance(cls, bigdl_type)\r\n     49             with cls._lock:\r\n     50                 if not cls._instance:\r\n---> 51                     cls._instance = cls(bigdl_type)\r\n     52         return cls._instance\r\n     53 \r\n\r\n/tmp/spark-21b61136-a031-4bc7-8e5b-032bbd07f63c/userFiles-d6fa7616-6c6e-445d-8233-153c1278d34d/bigdl-0.3.0-SNAPSHOT-python-api.zip/bigdl/util/common.py in __init__(self, bigdl_type)\r\n     67 \r\n     68     def __init__(self, bigdl_type):\r\n---> 69         sc = get_spark_context()\r\n     70         jclass = getattr(sc._jvm, JavaCreator.get_creator_class())\r\n     71         if bigdl_type == \"float\":\r\n\r\n/tmp/spark-21b61136-a031-4bc7-8e5b-032bbd07f63c/userFiles-d6fa7616-6c6e-445d-8233-153c1278d34d/bigdl-0.3.0-SNAPSHOT-python-api.zip/bigdl/util/common.py in get_spark_context(conf)\r\n    436     \"\"\"\r\n    437     if hasattr(SparkContext, \"getOrCreate\"):\r\n--> 438         return SparkContext.getOrCreate(conf=conf or create_spark_conf())\r\n    439     else:\r\n    440         # Might have threading issue but we cann't add _lock here\r\n\r\n/tmp/spark-21b61136-a031-4bc7-8e5b-032bbd07f63c/userFiles-d6fa7616-6c6e-445d-8233-153c1278d34d/bigdl-0.3.0-SNAPSHOT-python-api.zip/bigdl/util/common.py in create_spark_conf()\r\n    421 \r\n    422 def create_spark_conf():\r\n--> 423     bigdl_conf = get_bigdl_conf()\r\n    424     sparkConf = SparkConf()\r\n    425     sparkConf.setAll(bigdl_conf.items())\r\n\r\n/tmp/spark-21b61136-a031-4bc7-8e5b-032bbd07f63c/userFiles-d6fa7616-6c6e-445d-8233-153c1278d34d/bigdl-0.3.0-SNAPSHOT-python-api.zip/bigdl/util/common.py in get_bigdl_conf()\r\n    401             import zipfile\r\n    402             with zipfile.ZipFile(p, 'r') as zip_conf:\r\n--> 403                 content = zip_conf.read(bigdl_conf_file)\r\n    404                 if sys.version_info >= (3,):\r\n    405                     content = str(content, 'latin-1')\r\n\r\n/usr/lib/python2.7/zipfile.pyc in read(self, name, pwd)\r\n    933     def read(self, name, pwd=None):\r\n    934         \"\"\"Return file bytes (as a string) for name.\"\"\"\r\n--> 935         return self.open(name, \"r\", pwd).read()\r\n    936 \r\n    937     def open(self, name, mode=\"r\", pwd=None):\r\n\r\n/usr/lib/python2.7/zipfile.pyc in open(self, name, mode, pwd)\r\n    959             else:\r\n    960                 # Get info object for name\r\n--> 961                 zinfo = self.getinfo(name)\r\n    962 \r\n    963             zef_file.seek(zinfo.header_offset, 0)\r\n\r\n/usr/lib/python2.7/zipfile.pyc in getinfo(self, name)\r\n    907         if info is None:\r\n    908             raise KeyError(\r\n--> 909\r\n    910 \r\n    911         return info\r\n\r\nKeyError: \"There is no item named 'spark-bigdl.conf' in the archive\" \r\n```"}}
{"actor":{"display_login":"shane-huang"}}