{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/19524","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/19524/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/19524/comments","events_url":"https://api.github.com/repos/apache/spark/issues/19524/events","html_url":"https://github.com/apache/spark/pull/19524","id":266339422,"number":19524,"title":"[SPARK-22302][INFRA] Remove manual backports for subprocess and print explicit message for < Python 2.7","user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars0.githubusercontent.com/u/6477701?v=4","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-10-18T02:22:51Z","updated_at":"2017-10-18T06:47:07Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/19524","html_url":"https://github.com/apache/spark/pull/19524","diff_url":"https://github.com/apache/spark/pull/19524.diff","patch_url":"https://github.com/apache/spark/pull/19524.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nSeems there was a mistake - missing import for `subprocess.call`, while refactoring this script a long ago, which should be used for backports of some missing functions in `subprocess`, specifically in < Python 2.7.\r\n\r\nReproduction is:\r\n\r\n```\r\ncd dev && python2.6\r\n```\r\n\r\n```\r\n>>> from sparktestsupport import shellutils\r\n>>> shellutils.subprocess_check_call(\"ls\")\r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 1, in <module>\r\n  File \"sparktestsupport/shellutils.py\", line 46, in subprocess_check_call\r\n    retcode = call(*popenargs, **kwargs)\r\nNameError: global name 'call' is not defined\r\n```\r\n\r\nFor Jenkins logs, please see https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3950/console\r\n\r\nSince we dropped the Python 2.6.x support, looks better we remove those workarounds and print out explicit error messages in order to reduce the efforts to find out the root causes for such cases, for example, `https://github.com/apache/spark/pull/19513#issuecomment-337406734`.\r\n\r\n## How was this patch tested?\r\n\r\nManually tested:\r\n\r\n\r\n```\r\ncd dev && python2.6\r\n```\r\n\r\n```\r\n>>> from sparktestsupport import shellutils\r\n[error] Python versions prior to 2.7 are not supported.\r\n```\r\n\r\n```\r\ncd dev && python2.7\r\n```\r\n\r\n```\r\n>>> from sparktestsupport import shellutils\r\n>>> shellutils.subprocess_check_output(\"ls\")\r\n'README.md\\nappveyor-guide.md\\nappveyor-install-dependencies.ps1\\nchange-scala-version.sh\\ncheck-license\\ncheckstyle-suppressions.xml\\ncheckstyle.xml\\ncreate-release\\ndeps\\ngithub_jira_sync.py\\ngithub_jira_sync.pyc\\nlint-java\\nlint-python\\nlint-r\\nlint-r-report.log\\nlint-r.R\\nlint-scala\\nmake-distribution.sh\\nmerge_spark_pr.py\\nmerge_spark_pr.pyc\\nmerge_spark_pr_jira.py\\nmerge_spark_pr_jira.pyc\\nmima\\npep8-1.7.0.py\\npep8-1.7.0.pyc\\npip-sanity-check.py\\npip-sanity-check.pyc\\nrequirements.txt\\nrun-pip-tests\\nrun-tests\\nrun-tests-jenkins\\nrun-tests-jenkins.py\\nrun-tests-jenkins.pyc\\nrun-tests.py\\nrun-tests.pyc\\nscalastyle\\nsparktestsupport\\ntest-dependencies.sh\\ntests\\ntox.ini\\n'\r\n>>> shellutils.subprocess_check_call(\"ls\")\r\nREADME.md                         checkstyle-suppressions.xml       github_jira_sync.pyc              lint-r.R                          merge_spark_pr_jira.py            pip-sanity-check.py               run-tests-jenkins                 scalastyle\r\nappveyor-guide.md                 checkstyle.xml                    lint-java                         lint-scala                        merge_spark_pr_jira.pyc           pip-sanity-check.pyc              run-tests-jenkins.py              sparktestsupport\r\nappveyor-install-dependencies.ps1 create-release                    lint-python                       make-distribution.sh              mima                              requirements.txt                  run-tests-jenkins.pyc             test-dependencies.sh\r\nchange-scala-version.sh           deps                              lint-r                            merge_spark_pr.py                 pep8-1.7.0.py                     run-pip-tests                     run-tests.py                      tests\r\ncheck-license                     github_jira_sync.py               lint-r-report.log                 merge_spark_pr.pyc                pep8-1.7.0.pyc                    run-tests                         run-tests.pyc                     tox.ini\r\n0\r\n```\r\n\r\n```\r\ncd dev && python3.6\r\n```\r\n\r\n```\r\n>>> from sparktestsupport import shellutils\r\n>>> shellutils.subprocess_check_output(\"ls\")\r\nb'README.md\\nappveyor-guide.md\\nappveyor-install-dependencies.ps1\\nchange-scala-version.sh\\ncheck-license\\ncheckstyle-suppressions.xml\\ncheckstyle.xml\\ncreate-release\\ndeps\\ngithub_jira_sync.py\\ngithub_jira_sync.pyc\\nlint-java\\nlint-python\\nlint-r\\nlint-r-report.log\\nlint-r.R\\nlint-scala\\nmake-distribution.sh\\nmerge_spark_pr.py\\nmerge_spark_pr.pyc\\nmerge_spark_pr_jira.py\\nmerge_spark_pr_jira.pyc\\nmima\\npep8-1.7.0.py\\npep8-1.7.0.pyc\\npip-sanity-check.py\\npip-sanity-check.pyc\\nrequirements.txt\\nrun-pip-tests\\nrun-tests\\nrun-tests-jenkins\\nrun-tests-jenkins.py\\nrun-tests-jenkins.pyc\\nrun-tests.py\\nrun-tests.pyc\\nscalastyle\\nsparktestsupport\\ntest-dependencies.sh\\ntests\\ntox.ini\\n'\r\n>>> shellutils.subprocess_check_call(\"ls\")\r\nREADME.md                         checkstyle-suppressions.xml       github_jira_sync.pyc              lint-r.R                          merge_spark_pr_jira.py            pip-sanity-check.py               run-tests-jenkins                 scalastyle\r\nappveyor-guide.md                 checkstyle.xml                    lint-java                         lint-scala                        merge_spark_pr_jira.pyc           pip-sanity-check.pyc              run-tests-jenkins.py              sparktestsupport\r\nappveyor-install-dependencies.ps1 create-release                    lint-python                       make-distribution.sh              mima                              requirements.txt                  run-tests-jenkins.pyc             test-dependencies.sh\r\nchange-scala-version.sh           deps                              lint-r                            merge_spark_pr.py                 pep8-1.7.0.py                     run-pip-tests                     run-tests.py                      tests\r\ncheck-license                     github_jira_sync.py               lint-r-report.log                 merge_spark_pr.pyc                pep8-1.7.0.pyc                    run-tests                         run-tests.pyc                     tox.ini\r\n0"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/337476305","html_url":"https://github.com/apache/spark/pull/19524#issuecomment-337476305","issue_url":"https://api.github.com/repos/apache/spark/issues/19524","id":337476305,"user":{"login":"holdenk","id":59893,"avatar_url":"https://avatars1.githubusercontent.com/u/59893?v=4","gravatar_id":"","url":"https://api.github.com/users/holdenk","html_url":"https://github.com/holdenk","followers_url":"https://api.github.com/users/holdenk/followers","following_url":"https://api.github.com/users/holdenk/following{/other_user}","gists_url":"https://api.github.com/users/holdenk/gists{/gist_id}","starred_url":"https://api.github.com/users/holdenk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/holdenk/subscriptions","organizations_url":"https://api.github.com/users/holdenk/orgs","repos_url":"https://api.github.com/users/holdenk/repos","events_url":"https://api.github.com/users/holdenk/events{/privacy}","received_events_url":"https://api.github.com/users/holdenk/received_events","type":"User","site_admin":false},"created_at":"2017-10-18T06:47:07Z","updated_at":"2017-10-18T06:47:07Z","author_association":"CONTRIBUTOR","body":"Looks reasonable, my only thing is putting in a forced exited inside of shellutil.py is sort of a random place to the version check."}}
{"actor":{"display_login":"holdenk"}}