{"push_id":1688802865,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"35378766ad7d3c494425a8781efe9cb9349732b7","before":"773754b6c1516c15b64846a00e491535cbcb1007","commits":[{"sha":"35378766ad7d3c494425a8781efe9cb9349732b7","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-20343][BUILD] Avoid Unidoc build only if Hadoop 2.6 is explicitly set in SBT build\n\n## What changes were proposed in this pull request?\n\nThis PR proposes two things as below:\n\n- Avoid Unidoc build only if Hadoop 2.6 is explicitly set in SBT build\n\n  Due to a different dependency resolution in SBT & Unidoc by an unknown reason, the documentation build fails on a specific machine & environment in Jenkins but it was unable to reproduce.\n\n  So, this PR just checks an environment variable `AMPLAB_JENKINS_BUILD_PROFILE` that is set in Hadoop 2.6 SBT build against branches on Jenkins, and then disables Unidoc build. **Note that PR builder will still build it with Hadoop 2.6 & SBT.**\n\n  ```\n  ========================================================================\n  Building Unidoc API Documentation\n  ========================================================================\n  [info] Building Spark unidoc (w/Hive 1.2.1) using SBT with these arguments:  -Phadoop-2.6 -Pmesos -Pkinesis-asl -Pyarn -Phive-thriftserver -Phive unidoc\n  Using /usr/java/jdk1.8.0_60 as default JAVA_HOME.\n  ...\n  ```\n\n  I checked the environment variables from the logs (first bit) as below:\n\n  - **spark-master-test-sbt-hadoop-2.6** (this one is being failed) - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.6/lastBuild/consoleFull\n\n  ```\n  JAVA_HOME=/usr/java/jdk1.8.0_60\n  JAVA_7_HOME=/usr/java/jdk1.7.0_79\n  SPARK_BRANCH=master\n  AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.6   <- I use this variable\n  AMPLAB_JENKINS=\"true\"\n  ```\n  - spark-master-test-sbt-hadoop-2.7 - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/lastBuild/consoleFull\n\n  ```\n  JAVA_HOME=/usr/java/jdk1.8.0_60\n  JAVA_7_HOME=/usr/java/jdk1.7.0_79\n  SPARK_BRANCH=master\n  AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.7\n  AMPLAB_JENKINS=\"true\"\n  ```\n\n  - spark-master-test-maven-hadoop-2.6 - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.6/lastBuild/consoleFull\n\n  ```\n  JAVA_HOME=/usr/java/jdk1.8.0_60\n  JAVA_7_HOME=/usr/java/jdk1.7.0_79\n  HADOOP_PROFILE=hadoop-2.6\n  HADOOP_VERSION=\n  SPARK_BRANCH=master\n  AMPLAB_JENKINS=\"true\"\n  ```\n\n  - spark-master-test-maven-hadoop-2.7 - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/lastBuild/consoleFull\n\n  ```\n  JAVA_HOME=/usr/java/jdk1.8.0_60\n  JAVA_7_HOME=/usr/java/jdk1.7.0_79\n  HADOOP_PROFILE=hadoop-2.7\n  HADOOP_VERSION=\n  SPARK_BRANCH=master\n  AMPLAB_JENKINS=\"true\"\n  ```\n\n  - PR builder - https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75843/consoleFull\n\n  ```\n  JENKINS_MASTER_HOSTNAME=amp-jenkins-master\n  JAVA_HOME=/usr/java/jdk1.8.0_60\n  JAVA_7_HOME=/usr/java/jdk1.7.0_79\n  ```\n\n  Assuming from other logs in branch-2.1\n\n    - SBT & Hadoop 2.6 against branch-2.1 https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.1-test-sbt-hadoop-2.6/lastBuild/consoleFull\n\n      ```\n      JAVA_HOME=/usr/java/jdk1.8.0_60\n      JAVA_7_HOME=/usr/java/jdk1.7.0_79\n      SPARK_BRANCH=branch-2.1\n      AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.6\n      AMPLAB_JENKINS=\"true\"\n      ```\n\n    - Maven & Hadoop 2.6 against branch-2.1 https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.1-test-maven-hadoop-2.6/lastBuild/consoleFull\n\n      ```\n      JAVA_HOME=/usr/java/jdk1.8.0_60\n      JAVA_7_HOME=/usr/java/jdk1.7.0_79\n      HADOOP_PROFILE=hadoop-2.6\n      HADOOP_VERSION=\n      SPARK_BRANCH=branch-2.1\n      AMPLAB_JENKINS=\"true\"\n      ```\n\n  We have been using the same convention for those variables. These are actually being used in `run-tests.py` script - here https://github.com/apache/spark/blob/master/dev/run-tests.py#L519-L520\n\n- Revert the previous try\n\n  After https://github.com/apache/spark/pull/17651, it seems the build still fails on SBT Hadoop 2.6 master.\n\n  I am unable to reproduce this - https://github.com/apache/spark/pull/17477#issuecomment-294094092 and the reviewer was too. So, this got merged as it looks the only way to verify this is to merge it currently (as no one seems able to reproduce this).\n\n## How was this patch tested?\n\nI only checked `is_hadoop_version_2_6 = os.environ.get(\"AMPLAB_JENKINS_BUILD_PROFILE\") == \"hadoop2.6\"` is working fine as expected as below:\n\n```python\n>>> import collections\n>>> os = collections.namedtuple('os', 'environ')(environ={\"AMPLAB_JENKINS_BUILD_PROFILE\": \"hadoop2.6\"})\n>>> print(not os.environ.get(\"AMPLAB_JENKINS_BUILD_PROFILE\") == \"hadoop2.6\")\nFalse\n>>> os = collections.namedtuple('os', 'environ')(environ={\"AMPLAB_JENKINS_BUILD_PROFILE\": \"hadoop2.7\"})\n>>> print(not os.environ.get(\"AMPLAB_JENKINS_BUILD_PROFILE\") == \"hadoop2.6\")\nTrue\n>>> os = collections.namedtuple('os', 'environ')(environ={})\n>>> print(not os.environ.get(\"AMPLAB_JENKINS_BUILD_PROFILE\") == \"hadoop2.6\")\nTrue\n```\n\nI tried many ways but I was unable to reproduce this in my local. Sean also tried the way I did but he was also unable to reproduce this.\n\nPlease refer the comments in https://github.com/apache/spark/pull/17477#issuecomment-294094092\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #17669 from HyukjinKwon/revert-SPARK-20343.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/35378766ad7d3c494425a8781efe9cb9349732b7"}]}
{"actor":{"display_login":"zzcclp"}}