{"push_id":2124593337,"size":9,"distinct_size":9,"ref":"refs/heads/master","head":"eaff295a232217c4424f2885303f9a127dea0422","before":"209b9361ac8a4410ff797cff1115e1888e2f7e66","commits":[{"sha":"176ae4d53e0269cfc2cfa62d3a2991e28f5a9182","author":{"name":"Xianyang Liu","email":"62e5f0555779cb6eaf0ebf5f3688cb278d7523c9@intel.com"},"message":"[MINOR][CORE] Using bufferedInputStream for dataDeserializeStream\n\n## What changes were proposed in this pull request?\n\nSmall fix. Using bufferedInputStream for dataDeserializeStream.\n\n## How was this patch tested?\n\nExisting UT.\n\nAuthor: Xianyang Liu <xianyang.liu@intel.com>\n\nCloses #19735 from ConeyLiu/smallfix.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/176ae4d53e0269cfc2cfa62d3a2991e28f5a9182"},{"sha":"f7534b37ee91be14e511ab29259c3f83c7ad50af","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-22487][SQL][FOLLOWUP] still keep spark.sql.hive.version\n\n## What changes were proposed in this pull request?\n\na followup of https://github.com/apache/spark/pull/19712 , adds back the `spark.sql.hive.version`, so that if users try to read this config, they can still get a default value instead of null.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #19719 from cloud-fan/minor.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/f7534b37ee91be14e511ab29259c3f83c7ad50af"},{"sha":"c8b7f97b8a58bf4a9f6e3a07dd6e5b0f646d8d99","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-22377][BUILD] Use /usr/sbin/lsof if lsof does not exists in release-build.sh\n\n## What changes were proposed in this pull request?\n\nThis PR proposes to use `/usr/sbin/lsof` if `lsof` is missing in the path to fix nightly snapshot jenkins jobs. Please refer https://github.com/apache/spark/pull/19359#issuecomment-340139557:\n\n> Looks like some of the snapshot builds are having lsof issues:\n>\n> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.1-maven-snapshots/182/console\n>\n>https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.2-maven-snapshots/134/console\n>\n>spark-build/dev/create-release/release-build.sh: line 344: lsof: command not found\n>usage: kill [ -s signal | -p ] [ -a ] pid ...\n>kill -l [ signal ]\n\nUp to my knowledge,  the full path of `lsof` is required for non-root user in few OSs.\n\n## How was this patch tested?\n\nManually tested as below:\n\n```bash\n#!/usr/bin/env bash\n\nLSOF=lsof\nif ! hash $LSOF 2>/dev/null; then\n  echo \"a\"\n  LSOF=/usr/sbin/lsof\nfi\n\n$LSOF -P | grep \"a\"\n```\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #19695 from HyukjinKwon/SPARK-22377.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/c8b7f97b8a58bf4a9f6e3a07dd6e5b0f646d8d99"},{"sha":"d8741b2b0fe8b8da74f120859e969326fb170629","author":{"name":"WeichenXu","email":"95cb2f32d8c46dd3b757f6f1e0703161946bc0ee@databricks.com"},"message":"[SPARK-21911][ML][FOLLOW-UP] Fix doc for parallel ML Tuning in PySpark\n\n## What changes were proposed in this pull request?\n\nFix doc issue mentioned here: https://github.com/apache/spark/pull/19122#issuecomment-340111834\n\n## How was this patch tested?\n\nN/A\n\nAuthor: WeichenXu <weichen.xu@databricks.com>\n\nCloses #19641 from WeichenXu123/fix_doc.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/d8741b2b0fe8b8da74f120859e969326fb170629"},{"sha":"673c67046598d33b9ecf864024ca7a937c1998d6","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-17310][SQL] Add an option to disable record-level filter in Parquet-side\n\n## What changes were proposed in this pull request?\n\nThere is a concern that Spark-side codegen row-by-row filtering might be faster than Parquet's one in general due to type-boxing and additional fuction calls which Spark's one tries to avoid.\n\nSo, this PR adds an option to disable/enable record-by-record filtering in Parquet side.\n\nIt sets the default to `false` to take the advantage of the improvement.\n\nThis was also discussed in https://github.com/apache/spark/pull/14671.\n## How was this patch tested?\n\nManually benchmarks were performed. I generated a billion (1,000,000,000) records and tested equality comparison concatenated with `OR`. This filter combinations were made from 5 to 30.\n\nIt seem indeed Spark-filtering is faster in the test case and the gap increased as the filter tree becomes larger.\n\nThe details are as below:\n\n**Code**\n\n``` scala\ntest(\"Parquet-side filter vs Spark-side filter - record by record\") {\n  withTempPath { path =>\n    val N = 1000 * 1000 * 1000\n    val df = spark.range(N).toDF(\"a\")\n    df.write.parquet(path.getAbsolutePath)\n\n    val benchmark = new Benchmark(\"Parquet-side vs Spark-side\", N)\n    Seq(5, 10, 20, 30).foreach { num =>\n      val filterExpr = (0 to num).map(i => s\"a = $i\").mkString(\" OR \")\n\n      benchmark.addCase(s\"Parquet-side filter - number of filters [$num]\", 3) { _ =>\n        withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> false.toString,\n          SQLConf.PARQUET_RECORD_FILTER_ENABLED.key -> true.toString) {\n\n          // We should strip Spark-side filter to compare correctly.\n          stripSparkFilter(\n            spark.read.parquet(path.getAbsolutePath).filter(filterExpr)).count()\n        }\n      }\n\n      benchmark.addCase(s\"Spark-side filter - number of filters [$num]\", 3) { _ =>\n        withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> false.toString,\n          SQLConf.PARQUET_RECORD_FILTER_ENABLED.key -> false.toString) {\n\n          spark.read.parquet(path.getAbsolutePath).filter(filterExpr).count()\n        }\n      }\n    }\n\n    benchmark.run()\n  }\n}\n```\n\n**Result**\n\n```\nParquet-side vs Spark-side:              Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nParquet-side filter - number of filters [5]      4268 / 4367        234.3           4.3       0.8X\nSpark-side filter - number of filters [5]      3709 / 3741        269.6           3.7       0.9X\nParquet-side filter - number of filters [10]      5673 / 5727        176.3           5.7       0.6X\nSpark-side filter - number of filters [10]      3588 / 3632        278.7           3.6       0.9X\nParquet-side filter - number of filters [20]      8024 / 8440        124.6           8.0       0.4X\nSpark-side filter - number of filters [20]      3912 / 3946        255.6           3.9       0.8X\nParquet-side filter - number of filters [30]    11936 / 12041         83.8          11.9       0.3X\nSpark-side filter - number of filters [30]      3929 / 3978        254.5           3.9       0.8X\n```\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #15049 from HyukjinKwon/SPARK-17310.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/673c67046598d33b9ecf864024ca7a937c1998d6"},{"sha":"11b60af737a04d931356aa74ebf3c6cf4a6b08d6","author":{"name":"Zhenhua Wang","email":"93943f521ced4ea3f21d64a737435e7a7e171ac6@huawei.com"},"message":"[SPARK-17074][SQL] Generate equi-height histogram in column statistics\n\n## What changes were proposed in this pull request?\n\nEqui-height histogram is effective in cardinality estimation, and more accurate than basic column stats (min, max, ndv, etc) especially in skew distribution. So we need to support it.\n\nFor equi-height histogram, all buckets (intervals) have the same height (frequency).\nIn this PR, we use a two-step method to generate an equi-height histogram:\n1. use `ApproximatePercentile` to get percentiles `p(0), p(1/n), p(2/n) ... p((n-1)/n), p(1)`;\n2. construct range values of buckets, e.g. `[p(0), p(1/n)], [p(1/n), p(2/n)] ... [p((n-1)/n), p(1)]`, and use `ApproxCountDistinctForIntervals` to count ndv in each bucket. Each bucket is of the form: `(lowerBound, higherBound, ndv)`.\n\n## How was this patch tested?\n\nAdded new test cases and modified some existing test cases.\n\nAuthor: Zhenhua Wang <wangzhenhua@huawei.com>\nAuthor: Zhenhua Wang <wzh_zju@163.com>\n\nCloses #19479 from wzhfy/generate_histogram.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/11b60af737a04d931356aa74ebf3c6cf4a6b08d6"},{"sha":"4741c07809393ab85be8b4a169d4ed3da93a4781","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-20648][CORE] Port JobsTab and StageTab to the new UI backend.\n\nThis change is a little larger because there's a whole lot of logic\nbehind these pages, all really tied to internal types and listeners,\nand some of that logic had to be implemented in the new listener and\nthe needed data exposed through the API types.\n\n- Added missing StageData and ExecutorStageSummary fields which are\n  used by the UI. Some json golden files needed to be updated to account\n  for new fields.\n\n- Save RDD graph data in the store. This tries to re-use existing types as\n  much as possible, so that the code doesn't need to be re-written. So it's\n  probably not very optimal.\n\n- Some old classes (e.g. JobProgressListener) still remain, since they're used\n  in other parts of the code; they're not used by the UI anymore, though, and\n  will be cleaned up in a separate change.\n\n- Save information about active pools in the store. This data is not really used\n  in the SHS, but it's not a lot of data so it's still recorded when replaying\n  applications.\n\n- Because the new store sorts things slightly differently from the previous\n  code, some json golden files had some elements within them shuffled around.\n\n- The retention unit test in UISeleniumSuite was disabled because the code\n  to throw away old stages / tasks hasn't been added yet.\n\n- The job description field in the API tries to follow the old behavior, which\n  makes it be empty most of the time, even though there's information to fill it\n  in. For stages, a new field was added to hold the description (which is basically\n  the job description), so that the UI can be rendered in the old way.\n\n- A new stage status (\"SKIPPED\") was added to account for the fact that the API\n  couldn't represent that state before. Without this, the stage would show up as\n  \"PENDING\" in the UI, which is now based on API types.\n\n- The API used to expose \"executorRunTime\" as the value of the task's duration,\n  which wasn't really correct (also because that value was easily available\n  from the metrics object); this change fixes that by storing the correct duration,\n  which also means a few expectation files needed to be updated to account for\n  the new durations and sorting differences due to the changed values.\n\n- Added changes to implement SPARK-20713 and SPARK-21922 in the new code.\n\nTested with existing unit tests (and by using the UI a lot).\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #19698 from vanzin/SPARK-20648.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/4741c07809393ab85be8b4a169d4ed3da93a4781"},{"sha":"0ffa7c488fa8156e2a1aa282e60b7c36b86d8af8","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-20652][SQL] Store SQL UI data in the new app status store.\n\nThis change replaces the SQLListener with a new implementation that\nsaves the data to the same store used by the SparkContext's status\nstore. For that, the types used by the old SQLListener had to be\nupdated a bit so that they're more serialization-friendly.\n\nThe interface for getting data from the store was abstracted into\na new class, SQLAppStatusStore (following the convention used in\ncore).\n\nAnother change is the way that the SQL UI hooks up into the core\nUI or the SHS. The old \"SparkHistoryListenerFactory\" was replaced\nwith a new \"AppStatePlugin\" that more explicitly differentiates\nbetween the two use cases: processing events, and showing the UI.\nBoth live apps and the SHS use this new API (previously, it was\nrestricted to the SHS).\n\nNote on the above: this causes a slight change of behavior for\nlive apps; the SQL tab will only show up after the first execution\nis started.\n\nThe metrics gathering code was re-worked a bit so that the types\nused are less memory hungry and more serialization-friendly. This\nreduces memory usage when using in-memory stores, and reduces load\ntimes when using disk stores.\n\nTested with existing and added unit tests. Note one unit test was\ndisabled because it depends on SPARK-20653, which isn't in yet.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #19681 from vanzin/SPARK-20652.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/0ffa7c488fa8156e2a1aa282e60b7c36b86d8af8"},{"sha":"eaff295a232217c4424f2885303f9a127dea0422","author":{"name":"Devaraj K","email":"b8ebf05c6cb8ebae39c552537bfd7d84ebc26f96@apache.org"},"message":"[SPARK-22519][YARN] Remove unnecessary stagingDirPath null check in ApplicationMaster.cleanupStagingDir()\n\n## What changes were proposed in this pull request?\nRemoved the unnecessary stagingDirPath null check in ApplicationMaster.cleanupStagingDir().\n\n## How was this patch tested?\nI verified with the existing test cases.\n\nAuthor: Devaraj K <devaraj@apache.org>\n\nCloses #19749 from devaraj-kavali/SPARK-22519.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/eaff295a232217c4424f2885303f9a127dea0422"}]}
{"actor":{"display_login":"srowen"}}