{"push_id":2022625601,"size":86,"distinct_size":86,"ref":"refs/heads/spark-4502-parquet_column_pruning","head":"88786d3eaf9d3a3d2c80c2235c5014074ade3dc1","before":"9fac4826f6289e9e8c02f3799912463bbc8ea046","commits":[{"sha":"ce6a71e013c403d0a3690cf823934530ce0ea5ef","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-22076][SQL] Expand.projections should not be a Stream\n\n## What changes were proposed in this pull request?\n\nSpark with Scala 2.10 fails with a group by cube:\n```\nspark.range(1).select($\"id\" as \"a\", $\"id\" as \"b\").write.partitionBy(\"a\").mode(\"overwrite\").saveAsTable(\"rollup_bug\")\nspark.sql(\"select 1 from rollup_bug group by rollup ()\").show\n```\n\nIt can be traced back to https://github.com/apache/spark/pull/15484 , which made `Expand.projections` a lazy `Stream` for group by cube.\n\nIn scala 2.10 `Stream` captures a lot of stuff, and in this case it captures the entire query plan which has some un-serializable parts.\n\nThis change is also good for master branch, to reduce the serialized size of `Expand.projections`.\n\n## How was this patch tested?\n\nmanually verified with Spark with Scala 2.10.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #19289 from cloud-fan/bug.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/ce6a71e013c403d0a3690cf823934530ce0ea5ef"},{"sha":"bb9c0697d509880d30c874d7f6384d7633aed061","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-18838][HOTFIX][YARN] Check internal context state before stopping it.\n\nThe live listener bus now cleans up after itself and releases listeners\nafter stopping, so code cannot get references to listeners after the\nSpark context is stopped.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #19297 from vanzin/SPARK-18838.hotfix.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/bb9c0697d509880d30c874d7f6384d7633aed061"},{"sha":"55d5fa79db883e4d93a9c102a94713c9d2d1fb55","author":{"name":"Devaraj K","email":"b8ebf05c6cb8ebae39c552537bfd7d84ebc26f96@apache.org"},"message":"[SPARK-21384][YARN] Spark + YARN fails with LocalFileSystem as default FS\n\n## What changes were proposed in this pull request?\n\nWhen the libraries temp directory(i.e. __spark_libs__*.zip dir) file system and staging dir(destination) file systems are the same then the __spark_libs__*.zip is not copying to the staging directory. But after making this decision the libraries zip file is getting deleted immediately and becoming unavailable for the Node Manager's localization.\n\nWith this change, client copies the files to remote always when the source scheme is \"file\".\n\n## How was this patch tested?\n\nI have verified it manually in yarn/cluster and yarn/client modes with hdfs and local file systems.\n\nAuthor: Devaraj K <devaraj@apache.org>\n\nCloses #19141 from devaraj-kavali/SPARK-21384.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/55d5fa79db883e4d93a9c102a94713c9d2d1fb55"},{"sha":"352bea5457bf77bb6d0e45a8c71be104bee8a586","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-22076][SQL][FOLLOWUP] Expand.projections should not be a Stream\n\n## What changes were proposed in this pull request?\n\nThis a follow-up of https://github.com/apache/spark/pull/19289 , we missed another place: `rollup`. `Seq.init.toSeq` also returns a `Stream`, we should fix it too.\n\n## How was this patch tested?\n\nmanually\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #19298 from cloud-fan/bug.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/352bea5457bf77bb6d0e45a8c71be104bee8a586"},{"sha":"1da5822e6a60e024ae31cd722ca04abb2e7aed43","author":{"name":"jerryshao","email":"4c4fc98b369ca68b84d880910d6c592827f07854@hortonworks.com"},"message":"[SPARK-21934][CORE] Expose Shuffle Netty memory usage to MetricsSystem\n\n## What changes were proposed in this pull request?\n\nThis is a followup work of SPARK-9104 to expose the Netty memory usage to MetricsSystem. Current the shuffle Netty memory usage of `NettyBlockTransferService` will be exposed, if using external shuffle, then the Netty memory usage of `ExternalShuffleClient` and `ExternalShuffleService` will be exposed instead. Currently I don't expose Netty memory usage of `YarnShuffleService`, because `YarnShuffleService` doesn't have `MetricsSystem` itself, and is better to connect to Hadoop's MetricsSystem.\n\n## How was this patch tested?\n\nManually verified in local cluster.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #19160 from jerryshao/SPARK-21934.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/1da5822e6a60e024ae31cd722ca04abb2e7aed43"},{"sha":"a8d9ec8a60f21abb520b9109b238f914d2449022","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-21780][R] Simpler Dataset.sample API in R\n\n## What changes were proposed in this pull request?\n\nThis PR make `sample(...)` able to omit `withReplacement` defaulting to `FALSE`.\n\nIn short, the following examples are allowed:\n\n```r\n> df <- createDataFrame(as.list(seq(10)))\n> count(sample(df, fraction=0.5, seed=3))\n[1] 4\n> count(sample(df, fraction=1.0))\n[1] 10\n```\n\nIn addition, this PR also adds some type checking logics as below:\n\n```r\n> sample(df, fraction = \"a\")\nError in sample(df, fraction = \"a\") :\n  fraction must be numeric; however, got character\n> sample(df, fraction = 1, seed = NULL)\nError in sample(df, fraction = 1, seed = NULL) :\n  seed must not be NULL or NA; however, got NULL\n> sample(df, list(1), 1.0)\nError in sample(df, list(1), 1) :\n  withReplacement must be logical; however, got list\n> sample(df, fraction = -1.0)\n...\nError in sample : illegal argument - requirement failed: Sampling fraction (-1.0) must be on interval [0, 1] without replacement\n```\n\n## How was this patch tested?\n\nManually tested, unit tests added in `R/pkg/tests/fulltests/test_sparkSQL.R`.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #19243 from HyukjinKwon/SPARK-21780.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/a8d9ec8a60f21abb520b9109b238f914d2449022"},{"sha":"1d1a09be9fe07244d7d12442f6105f823b260755","author":{"name":"Zhenhua Wang","email":"93943f521ced4ea3f21d64a737435e7a7e171ac6@huawei.com"},"message":"[SPARK-17997][SQL] Add an aggregation function for counting distinct values for multiple intervals\n\n## What changes were proposed in this pull request?\n\nThis work is a part of [SPARK-17074](https://issues.apache.org/jira/browse/SPARK-17074) to compute equi-height histograms. Equi-height histogram is an array of bins. A bin consists of two endpoints which form an interval of values and the ndv in that interval.\n\nThis PR creates a new aggregate function, given an array of endpoints, counting distinct values (ndv) in intervals among those endpoints.\n\nThis PR also refactors `HyperLogLogPlusPlus` by extracting a helper class `HyperLogLogPlusPlusHelper`, where the underlying HLLPP algorithm locates.\n\n## How was this patch tested?\n\nAdd new test cases.\n\nAuthor: Zhenhua Wang <wangzhenhua@huawei.com>\n\nCloses #15544 from wzhfy/countIntervals.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/1d1a09be9fe07244d7d12442f6105f823b260755"},{"sha":"1270e71753f40c353fb726a0a3d373d181aedb40","author":{"name":"Liang-Chi Hsieh","email":"9f98897f009ec1944a62ee7d389f9f5b1873efea@gmail.com"},"message":"[SPARK-22086][DOCS] Add expression description for CASE WHEN\n\n## What changes were proposed in this pull request?\n\nIn SQL conditional expressions, only CASE WHEN lacks for expression description. This patch fills the gap.\n\n## How was this patch tested?\n\nOnly documentation change.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #19304 from viirya/casewhen-doc.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/1270e71753f40c353fb726a0a3d373d181aedb40"},{"sha":"f10cbf17dc7ceb96982fcdc964c849336fb50deb","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-21977][HOTFIX] Adjust EnsureStatefulOpPartitioningSuite to use scalatest lifecycle normally instead of constructor\n\n## What changes were proposed in this pull request?\n\nAdjust EnsureStatefulOpPartitioningSuite to use scalatest lifecycle normally instead of constructor; fixes:\n\n```\n*** RUN ABORTED ***\n  org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true. The currently running SparkContext was created at:\norg.apache.spark.sql.streaming.EnsureStatefulOpPartitioningSuite.<init>(EnsureStatefulOpPartitioningSuite.scala:35)\n```\n\n## How was this patch tested?\n\nExisting tests\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #19306 from srowen/SPARK-21977.2.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/f10cbf17dc7ceb96982fcdc964c849336fb50deb"},{"sha":"b75bd1777496ce0354458bf85603a8087a6a0ff8","author":{"name":"Imran Rashid","email":"63f21f1cb7b694d385ee53376e18c9c9be6c6f98@cloudera.com"},"message":"[SPARK-21928][CORE] Set classloader on SerializerManager's private kryo\n\n## What changes were proposed in this pull request?\n\nWe have to make sure that SerializerManager's private instance of\nkryo also uses the right classloader, regardless of the current thread\nclassloader.  In particular, this fixes serde during remote cache\nfetches, as those occur in netty threads.\n\n## How was this patch tested?\n\nManual tests & existing suite via jenkins.  I haven't been able to reproduce this is in a unit test, because when a remote RDD partition can be fetched, there is a warning message and then the partition is just recomputed locally.  I manually verified the warning message is no longer present.\n\nAuthor: Imran Rashid <irashid@cloudera.com>\n\nCloses #19280 from squito/SPARK-21928_ser_classloader.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/b75bd1777496ce0354458bf85603a8087a6a0ff8"},{"sha":"f7ad0dbd512b00241337f15e1ed0a28a1d30711d","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[INFRA] Close stale PRs.\n\nCloses #19296\nCloses #19291","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/f7ad0dbd512b00241337f15e1ed0a28a1d30711d"},{"sha":"9cac249fd56cb001a723733eb0d6da606bd88190","author":{"name":"Liang-Chi Hsieh","email":"9f98897f009ec1944a62ee7d389f9f5b1873efea@gmail.com"},"message":"[SPARK-22088][SQL] Incorrect scalastyle comment causes wrong styles in stringExpressions\n\n## What changes were proposed in this pull request?\n\nThere is an incorrect `scalastyle:on` comment in `stringExpressions.scala` and causes the line size limit check ineffective in the file. There are many lines of code and comment which are more than 100 chars.\n\n## How was this patch tested?\n\nCode style change only.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #19305 from viirya/fix-wrong-style.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/9cac249fd56cb001a723733eb0d6da606bd88190"},{"sha":"b21b806ecc55f15575833c1e859c35ae391ff369","author":{"name":"Zheng RuiFeng","email":"688fbc7eb0ddb0dce0ccb704b29547c7b3cab719@foxmail.com"},"message":"[SPARK-22075][ML] GBTs unpersist datasets cached by Checkpointer\n\n## What changes were proposed in this pull request?\n`PeriodicRDDCheckpointer` will automatically persist the last 3 datasets called by `PeriodicRDDCheckpointer.update()`.\nIn GBTs, the last 3 intermediate rdds are still cached after `fit()`\n\n## How was this patch tested?\nexisting tests and local test in spark-shell\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #19288 from zhengruifeng/gbt_unpersist.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/b21b806ecc55f15575833c1e859c35ae391ff369"},{"sha":"a8a5cd24e2b9255acbc1e946f2946928144568ae","author":{"name":"Zheng RuiFeng","email":"688fbc7eb0ddb0dce0ccb704b29547c7b3cab719@foxmail.com"},"message":"[SPARK-22009][ML] Using treeAggregate improve some algs\n\n## What changes were proposed in this pull request?\n\nI test on a dataset of about 13M instances, and found that using `treeAggregate` give a speedup in following algs:\n\n|Algs| SpeedUp |\n|------|-----------|\n|OneHotEncoder| 5% |\n|StatFunctions.calculateCov| 7% |\n|StatFunctions.multipleApproxQuantiles|  9% |\n|RegressionEvaluator| 8% |\n\n## How was this patch tested?\nexisting tests\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #19232 from zhengruifeng/use_treeAggregate.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/a8a5cd24e2b9255acbc1e946f2946928144568ae"},{"sha":"f32a8425051eabdef2d69002cfc843c01d98df0d","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-22053][SS] Stream-stream inner join in Append Mode\n\n## What changes were proposed in this pull request?\n\n#### Architecture\nThis PR implements stream-stream inner join using a two-way symmetric hash join. At a high level, we want to do the following.\n\n1. For each stream, we maintain the past rows as state in State Store.\n  - For each joining key, there can be multiple rows that have been received.\n  - So, we have to effectively maintain a key-to-list-of-values multimap as state for each stream.\n2. In each batch, for each input row in each stream\n  - Look up the other streams state to see if there are matching rows, and output them if they satisfy the joining condition\n  - Add the input row to corresponding stream’s state.\n  - If the data has a timestamp/window column with watermark, then we will use that to calculate the threshold for keys that are required to buffered for future matches and drop the rest from the state.\n\nCleaning up old unnecessary state rows depends completely on whether watermark has been defined and what are join conditions. We definitely want to support state clean up two types of queries that are likely to be common.\n\n- Queries to time range conditions - E.g. `SELECT * FROM leftTable, rightTable ON leftKey = rightKey AND leftTime > rightTime - INTERVAL 8 MINUTES AND leftTime < rightTime + INTERVAL 1 HOUR`\n- Queries with windows as the matching key - E.g. `SELECT * FROM leftTable, rightTable ON leftKey = rightKey AND window(leftTime, \"1 hour\") = window(rightTime, \"1 hour\")` (pseudo-SQL)\n\n#### Implementation\nThe stream-stream join is primarily implemented in three classes\n- `StreamingSymmetricHashJoinExec` implements the above symmetric join algorithm.\n- `SymmetricsHashJoinStateManagers` manages the streaming state for the join. This essentially is a fault-tolerant key-to-list-of-values multimap built on the StateStore APIs. `StreamingSymmetricHashJoinExec` instantiates two such managers, one for each join side.\n- `StreamingSymmetricHashJoinExecHelper` is a helper class to extract threshold for the state based on the join conditions and the event watermark.\n\nRefer to the scaladocs class for more implementation details.\n\nBesides the implementation of stream-stream inner join SparkPlan. Some additional changes are\n- Allowed inner join in append mode in UnsupportedOperationChecker\n- Prevented stream-stream join on an empty batch dataframe to be collapsed by the optimizer\n\n## How was this patch tested?\n- New tests in StreamingJoinSuite\n- Updated tests UnsupportedOperationSuite\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #19271 from tdas/SPARK-22053.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/f32a8425051eabdef2d69002cfc843c01d98df0d"},{"sha":"fedf6961be4e99139eb7ab08d5e6e29187ea5ccf","author":{"name":"Shixiong Zhu","email":"c8f54caa5df4e9592a01136ca859e0af6b5a2c2a@gmail.com"},"message":"[SPARK-22094][SS] processAllAvailable should check the query state\n\n## What changes were proposed in this pull request?\n\n`processAllAvailable` should also check the query state and if the query is stopped, it should return.\n\n## How was this patch tested?\n\nThe new unit test.\n\nAuthor: Shixiong Zhu <zsxwing@gmail.com>\n\nCloses #19314 from zsxwing/SPARK-22094.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/fedf6961be4e99139eb7ab08d5e6e29187ea5ccf"},{"sha":"5ac96854cc6186fa2dad602d0906ff2705e3f610","author":{"name":"Marco Gaido","email":"737a3c370a64b00d8905aa495392d09750dfa37c@hortonworks.com"},"message":"[SPARK-21981][PYTHON][ML] Added Python interface for ClusteringEvaluator\n\n## What changes were proposed in this pull request?\n\nAdded Python interface for ClusteringEvaluator\n\n## How was this patch tested?\n\nManual test, eg. the example Python code in the comments.\n\ncc yanboliang\n\nAuthor: Marco Gaido <mgaido@hortonworks.com>\nAuthor: Marco Gaido <marcogaido91@gmail.com>\n\nCloses #19204 from mgaido91/SPARK-21981.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/5ac96854cc6186fa2dad602d0906ff2705e3f610"},{"sha":"5960686e791b5d6642a30c43c1de61e96e594a5e","author":{"name":"maryannxue","email":"016ed15fd34955cd77bc1b92456120bf8d3c1ff4@gmail.com"},"message":"[SPARK-21998][SQL] SortMergeJoinExec did not calculate its outputOrdering correctly during physical planning\n\n## What changes were proposed in this pull request?\n\nRight now the calculation of SortMergeJoinExec's outputOrdering relies on the fact that its children have already been sorted on the join keys, while this is often not true until EnsureRequirements has been applied. So we ended up not getting the correct outputOrdering during physical planning stage before Sort nodes are added to the children.\n\nFor example, J = {A join B on key1 = key2}\n1. if A is NOT ordered on key1 ASC, J's outputOrdering should include \"key1 ASC\"\n2. if A is ordered on key1 ASC, J's outputOrdering should include \"key1 ASC\"\n3. if A is ordered on key1 ASC, with sameOrderExp=c1, J's outputOrdering should include \"key1 ASC, sameOrderExp=c1\"\n\nSo to fix this I changed the  behavior of <code>getKeyOrdering(keys, childOutputOrdering)</code> to:\n1. If the childOutputOrdering satisfies (is a superset of) the required child ordering => childOutputOrdering\n2. Otherwise => required child ordering\n\nIn addition, I organized the logic for deciding the relationship between two orderings into SparkPlan, so that it can be reused by EnsureRequirements and SortMergeJoinExec, and potentially other classes.\n\n## How was this patch tested?\n\nAdded new test cases.\nPassed all integration tests.\n\nAuthor: maryannxue <maryann.xue@gmail.com>\n\nCloses #19281 from maryannxue/spark-21998.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/5960686e791b5d6642a30c43c1de61e96e594a5e"},{"sha":"8f130ad40178e35fecb3f2ba4a61ad23e6a90e3d","author":{"name":"Holden Karau","email":"0dd5affe763abbc7f4de3fb3797050edd8c481d3@us.ibm.com"},"message":"[SPARK-22072][SPARK-22071][BUILD] Improve release build scripts\n\n## What changes were proposed in this pull request?\n\nCheck JDK version (with javac) and use SPARK_VERSION for publish-release\n\n## How was this patch tested?\n\nManually tried local build with wrong JDK / JAVA_HOME & built a local release (LFTP disabled)\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #19312 from holdenk/improve-release-scripts-r2.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/8f130ad40178e35fecb3f2ba4a61ad23e6a90e3d"},{"sha":"27fc536d9a54eccef7d1cbbe2a6a008043d62ba4","author":{"name":"Bryan Cutler","email":"62bf89cc0e70e22c1fc5a3d8c3b77c0895eb33c3@gmail.com"},"message":"[SPARK-21190][PYSPARK] Python Vectorized UDFs\n\nThis PR adds vectorized UDFs to the Python API\n\n**Proposed API**\nIntroduce a flag to turn on vectorization for a defined UDF, for example:\n\n```\npandas_udf(DoubleType())\ndef plus(a, b)\n    return a + b\n```\nor\n\n```\nplus = pandas_udf(lambda a, b: a + b, DoubleType())\n```\nUsage is the same as normal UDFs\n\n0-parameter UDFs\npandas_udf functions can declare an optional `**kwargs` and when evaluated, will contain a key \"size\" that will give the required length of the output.  For example:\n\n```\npandas_udf(LongType())\ndef f0(**kwargs):\n    return pd.Series(1).repeat(kwargs[\"size\"])\n\ndf.select(f0())\n```\n\nAdded new unit tests in pyspark.sql that are enabled if pyarrow and Pandas are available.\n\n- [x] Fix support for promoted types with null values\n- [ ] Discuss 0-param UDF API (use of kwargs)\n- [x] Add tests for chained UDFs\n- [ ] Discuss behavior when pyarrow not installed / enabled\n- [ ] Cleanup pydoc and add user docs\n\nAuthor: Bryan Cutler <cutlerb@gmail.com>\nAuthor: Takuya UESHIN <ueshin@databricks.com>\n\nCloses #18659 from BryanCutler/arrow-vectorized-udfs-SPARK-21404.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/27fc536d9a54eccef7d1cbbe2a6a008043d62ba4"}]}
{"actor":{"display_login":"mallman"}}