{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/18981","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/18981/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/18981/comments","events_url":"https://api.github.com/repos/apache/spark/issues/18981/events","html_url":"https://github.com/apache/spark/pull/18981","id":251097152,"number":18981,"title":"Fixed pandoc dependency issue in python/setup.py","user":{"login":"dusktreader","id":713676,"avatar_url":"https://avatars0.githubusercontent.com/u/713676?v=4","gravatar_id":"","url":"https://api.github.com/users/dusktreader","html_url":"https://github.com/dusktreader","followers_url":"https://api.github.com/users/dusktreader/followers","following_url":"https://api.github.com/users/dusktreader/following{/other_user}","gists_url":"https://api.github.com/users/dusktreader/gists{/gist_id}","starred_url":"https://api.github.com/users/dusktreader/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dusktreader/subscriptions","organizations_url":"https://api.github.com/users/dusktreader/orgs","repos_url":"https://api.github.com/users/dusktreader/repos","events_url":"https://api.github.com/users/dusktreader/events{/privacy}","received_events_url":"https://api.github.com/users/dusktreader/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-08-17T23:09:36Z","updated_at":"2017-08-19T00:10:21Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/18981","html_url":"https://github.com/apache/spark/pull/18981","diff_url":"https://github.com/apache/spark/pull/18981.diff","patch_url":"https://github.com/apache/spark/pull/18981.patch"},"body":"## Problem Description\r\n\r\nWhen pyspark is listed as a dependency of another package, installing\r\nthe other package will cause an install failure in pyspark. When the\r\nother package is being installed, pyspark's setup_requires requirements\r\nare installed including pypandoc. Thus, the exception handling on\r\nsetup.py:152 does not work because the pypandoc module is indeed\r\navailable. However, the pypandoc.convert() function fails if pandoc\r\nitself is not installed (in our use cases it is not). This raises an\r\nOSError that is not handled, and setup fails.\r\n\r\nThe following is a sample failure:\r\n```\r\n$ which pandoc\r\n$ pip freeze | grep pypandoc\r\npypandoc==1.4\r\n$ pip install pyspark\r\nCollecting pyspark\r\n  Downloading pyspark-2.2.0.post0.tar.gz (188.3MB)\r\n    100% |████████████████████████████████| 188.3MB 16.8MB/s\r\n    Complete output from command python setup.py egg_info:\r\n    Maybe try:\r\n\r\n        sudo apt-get install pandoc\r\n    See http://johnmacfarlane.net/pandoc/installing.html\r\n    for installation options\r\n    ---------------------------------------------------------------\r\n\r\n    Traceback (most recent call last):\r\n      File \"<string>\", line 1, in <module>\r\n      File \"/tmp/pip-build-mfnizcwa/pyspark/setup.py\", line 151, in <module>\r\n        long_description = pypandoc.convert('README.md', 'rst')\r\n      File \"/home/tbeck/.virtualenvs/cem/lib/python3.5/site-packages/pypandoc/__init__.py\", line 69, in convert\r\n        outputfile=outputfile, filters=filters)\r\n      File \"/home/tbeck/.virtualenvs/cem/lib/python3.5/site-packages/pypandoc/__init__.py\", line 260, in _convert_input\r\n        _ensure_pandoc_path()\r\n      File \"/home/tbeck/.virtualenvs/cem/lib/python3.5/site-packages/pypandoc/__init__.py\", line 544, in _ensure_pandoc_path\r\n        raise OSError(\"No pandoc was found: either install pandoc and add it\\n\"\r\n    OSError: No pandoc was found: either install pandoc and add it\r\n    to your PATH or or call pypandoc.download_pandoc(...) or\r\n    install pypandoc wheels with included pandoc.\r\n\r\n    ----------------------------------------\r\nCommand \"python setup.py egg_info\" failed with error code 1 in /tmp/pip-build-mfnizcwa/pyspark/\r\n```\r\n\r\n## What changes were proposed in this pull request?\r\n\r\nThis change simply adds an additional exception handler for the OSError\r\nthat is raised. This allows pyspark to be installed client-side without requiring pandoc to be installed.\r\n\r\n## How was this patch tested?\r\n\r\nI tested this by building a wheel package of pyspark with the change applied. Then, in a clean virtual environment with pypandoc installed but pandoc not available on the system, I installed pyspark from the wheel.\r\n\r\nHere is the output\r\n\r\n```\r\n$ pip freeze | grep pypandoc\r\npypandoc==1.4\r\n$ which pandoc\r\n$ pip install --no-cache-dir ../spark/python/dist/pyspark-2.3.0.dev0-py2.py3-none-any.whl \r\nProcessing /home/tbeck/work/spark/python/dist/pyspark-2.3.0.dev0-py2.py3-none-any.whl\r\nRequirement already satisfied: py4j==0.10.6 in /home/tbeck/.virtualenvs/cem/lib/python3.5/site-packages (from pyspark==2.3.0.dev0)\r\nInstalling collected packages: pyspark\r\nSuccessfully installed pyspark-2.3.0.dev0\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/323486256","html_url":"https://github.com/apache/spark/pull/18981#issuecomment-323486256","issue_url":"https://api.github.com/repos/apache/spark/issues/18981","id":323486256,"user":{"login":"holdenk","id":59893,"avatar_url":"https://avatars1.githubusercontent.com/u/59893?v=4","gravatar_id":"","url":"https://api.github.com/users/holdenk","html_url":"https://github.com/holdenk","followers_url":"https://api.github.com/users/holdenk/followers","following_url":"https://api.github.com/users/holdenk/following{/other_user}","gists_url":"https://api.github.com/users/holdenk/gists{/gist_id}","starred_url":"https://api.github.com/users/holdenk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/holdenk/subscriptions","organizations_url":"https://api.github.com/users/holdenk/orgs","repos_url":"https://api.github.com/users/holdenk/repos","events_url":"https://api.github.com/users/holdenk/events{/privacy}","received_events_url":"https://api.github.com/users/holdenk/received_events","type":"User","site_admin":false},"created_at":"2017-08-19T00:10:21Z","updated_at":"2017-08-19T00:10:21Z","body":"Thanks for fixing this @dusktreader :) OSError seems to mostly occur when pandoc is not installed.\r\n\r\nI think automatically attempting to install pandoc is more likely to lead us into difficulty rather than skipping it (since otherwise in part pypandoc would just do this automatically).\r\n\r\nJenkins, OK to Test."}}
{"actor":{"display_login":"holdenk"}}