{"push_id":1533527547,"size":280,"distinct_size":280,"ref":"refs/heads/master","head":"b0985764f00acea97df7399a6b337262fc97f5ee","before":"ba4468bb24f2e13b98e7b66b44203c20393b8ab8","commits":[{"sha":"b7650f11c7afbdffc6f5caaafb5dcfd54f7a25ff","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-18947][SQL] SQLContext.tableNames should not call Catalog.listTables\n\n## What changes were proposed in this pull request?\n\nIt's a huge waste to call `Catalog.listTables` in `SQLContext.tableNames`, which only need the table names, while `Catalog.listTables` will get the table metadata for each table name.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #16352 from cloud-fan/minor.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/b7650f11c7afbdffc6f5caaafb5dcfd54f7a25ff"},{"sha":"1a64388973711b4e567f25fa33d752066a018b49","author":{"name":"Yin Huai","email":"7e8362bd012819bee68b9375d9cefa24afbef722@databricks.com"},"message":"[SPARK-18951] Upgrade com.thoughtworks.paranamer/paranamer to 2.6\n\n## What changes were proposed in this pull request?\nI recently hit a bug of com.thoughtworks.paranamer/paranamer, which causes jackson fail to handle byte array defined in a case class. Then I find https://github.com/FasterXML/jackson-module-scala/issues/48, which suggests that it is caused by a bug in paranamer. Let's upgrade paranamer. Since we are using jackson 2.6.5 and jackson-module-paranamer 2.6.5 use com.thoughtworks.paranamer/paranamer 2.6, I suggests that we upgrade paranamer to 2.6.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #16359 from yhuai/SPARK-18951.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/1a64388973711b4e567f25fa33d752066a018b49"},{"sha":"607a1e63dbc9269b806a9f537e1d041029333cdd","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-18894][SS] Fix event time watermark delay threshold specified in months or years\n\n## What changes were proposed in this pull request?\n\nTwo changes\n- Fix how delays specified in months and years are translated to milliseconds\n- Following up on #16258, not show watermark when there is no watermarking in the query\n\n## How was this patch tested?\nUpdated and new unit tests\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #16304 from tdas/SPARK-18834-1.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/607a1e63dbc9269b806a9f537e1d041029333cdd"},{"sha":"ccfe60a8304871779ff1b31b8c2d724f59d5b2af","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-18031][TESTS] Fix flaky test ExecutorAllocationManagerSuite.basic functionality\n\n## What changes were proposed in this pull request?\n\nThe failure is because in `test(\"basic functionality\")`, it doesn't block until `ExecutorAllocationManager.manageAllocation` is called. This PR just adds StreamManualClock to allow the tests to block on expected wait time to make the test deterministic.\n\n## How was this patch tested?\n\nJenkins\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #16321 from zsxwing/SPARK-18031.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/ccfe60a8304871779ff1b31b8c2d724f59d5b2af"},{"sha":"078c71c2dcbb1470d22f8eb8138fb17e3d7c2414","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-18954][TESTS] Fix flaky test: o.a.s.streaming.BasicOperationsSuite rdd cleanup - map and window\n\n## What changes were proposed in this pull request?\n\nThe issue in this test is the cleanup of RDDs may not be able to finish before stopping StreamingContext. This PR basically just puts the assertions into `eventually` and runs it before stopping StreamingContext.\n\n## How was this patch tested?\n\nJenkins\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #16362 from zsxwing/SPARK-18954.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/078c71c2dcbb1470d22f8eb8138fb17e3d7c2414"},{"sha":"354e936187708a404c0349e3d8815a47953123ec","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-18775][SQL] Limit the max number of records written per file\n\n## What changes were proposed in this pull request?\nCurrently, Spark writes a single file out per task, sometimes leading to very large files. It would be great to have an option to limit the max number of records written per file in a task, to avoid humongous files.\n\nThis patch introduces a new write config option `maxRecordsPerFile` (default to a session-wide setting `spark.sql.files.maxRecordsPerFile`) that limits the max number of records written to a single file. A non-positive value indicates there is no limit (same behavior as not having this flag).\n\n## How was this patch tested?\nAdded test cases in PartitionedWriteSuite for both dynamic partition insert and non-dynamic partition insert.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #16204 from rxin/SPARK-18775.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/354e936187708a404c0349e3d8815a47953123ec"},{"sha":"95efc895e929701a605313b87ad0cd91edee2f81","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-18588][SS][KAFKA] Create a new KafkaConsumer when error happens to fix the flaky test\n\n## What changes were proposed in this pull request?\n\nWhen KafkaSource fails on Kafka errors, we should create a new consumer to retry rather than using the existing broken one because it's possible that the broken one will fail again.\n\nThis PR also assigns a new group id to the new created consumer for a possible race condition:  the broken consumer cannot talk with the Kafka cluster in `close` but the new consumer can talk to Kafka cluster. I'm not sure if this will happen or not. Just for safety to avoid that the Kafka cluster thinks there are two consumers with the same group id in a short time window. (Note: CachedKafkaConsumer doesn't need this fix since `assign` never uses the group id.)\n\n## How was this patch tested?\n\nIn https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70370/console , it ran this flaky test 120 times and all passed.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #16282 from zsxwing/kafka-fix.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/95efc895e929701a605313b87ad0cd91edee2f81"},{"sha":"afd9bc1d8a85adf88c412d8bc75e46e7ecb4bcdd","author":{"name":"Ryan Williams","email":"637449b15902c748deeeab1d4dc8444f2edc120b@gmail.com"},"message":"[SPARK-17807][CORE] split test-tags into test-JAR\n\nRemove spark-tag's compile-scope dependency (and, indirectly, spark-core's compile-scope transitive-dependency) on scalatest by splitting test-oriented tags into spark-tags' test JAR.\n\nAlternative to #16303.\n\nAuthor: Ryan Williams <ryan.blake.williams@gmail.com>\n\nCloses #16311 from ryan-williams/tt.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/afd9bc1d8a85adf88c412d8bc75e46e7ecb4bcdd"},{"sha":"83a6ace0d1be44f70e768348ae6688798c84343e","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-18234][SS] Made update mode public\n\n## What changes were proposed in this pull request?\n\nMade update mode public. As part of that here are the changes.\n- Update DatastreamWriter to accept \"update\"\n- Changed package of InternalOutputModes from o.a.s.sql to o.a.s.sql.catalyst\n- Added update mode state removing with watermark to StateStoreSaveExec\n\n## How was this patch tested?\n\nAdded new tests in changed modules\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #16360 from tdas/SPARK-18234.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/83a6ace0d1be44f70e768348ae6688798c84343e"},{"sha":"b41ec997786e2be42a8a2a182212a610d08b221b","author":{"name":"Takeshi YAMAMURO","email":"c77105bb36ce8e17a5fe8a8e3953cfa3aa730380@gmail.com"},"message":"[SPARK-18528][SQL] Fix a bug to initialise an iterator of aggregation buffer\n\n## What changes were proposed in this pull request?\nThis pr is to fix an `NullPointerException` issue caused by a following `limit + aggregate` query;\n```\nscala> val df = Seq((\"a\", 1), (\"b\", 2), (\"c\", 1), (\"d\", 5)).toDF(\"id\", \"value\")\nscala> df.limit(2).groupBy(\"id\").count().show\nWARN TaskSetManager: Lost task 0.0 in stage 9.0 (TID 8204, lvsp20hdn012.stubprod.com): java.lang.NullPointerException\nat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.agg_doAggregateWithKeys$(Unknown Source)\nat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\n```\nThe root culprit is that [`$doAgg()`](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala#L596) skips an initialization of [the buffer iterator](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala#L603); `BaseLimitExec` sets `stopEarly=true` and `$doAgg()` exits in the middle without the initialization.\n\n## How was this patch tested?\nAdded a test to check if no exception happens for limit + aggregates in `DataFrameAggregateSuite.scala`.\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #15980 from maropu/SPARK-18528.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/b41ec997786e2be42a8a2a182212a610d08b221b"},{"sha":"7e8994ffd3d646adb0a769229637931e43cd12b0","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-18903][SPARKR] Add API to get SparkUI URL\n\n## What changes were proposed in this pull request?\n\nAPI for SparkUI URL from SparkContext\n\n## How was this patch tested?\n\nmanual, unit tests\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #16367 from felixcheung/rwebui.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/7e8994ffd3d646adb0a769229637931e43cd12b0"},{"sha":"afe36516e4b4031196ee2e0a04980ac49208ea6b","author":{"name":"Burak Yavuz","email":"39bd9266538731f16447e7d968c0ba25275ede1e@gmail.com"},"message":"[FLAKY-TEST] InputStreamsSuite.socket input stream\n\n## What changes were proposed in this pull request?\n\nhttps://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.streaming.InputStreamsSuite&test_name=socket+input+stream\n\n## How was this patch tested?\n\nTested 2,000 times.\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #16343 from brkyvz/sock.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/afe36516e4b4031196ee2e0a04980ac49208ea6b"},{"sha":"e1b43dc45b136a89f105c04c3074233f52568152","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[BUILD] make-distribution should find JAVA_HOME for non-RHEL systems\n\n## What changes were proposed in this pull request?\n\nmake-distribution.sh should find JAVA_HOME for Ubuntu, Mac and other non-RHEL systems\n\n## How was this patch tested?\n\nManually\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #16363 from felixcheung/buildjava.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/e1b43dc45b136a89f105c04c3074233f52568152"},{"sha":"ff7d82a207e8bef7779c27378f7a50a138627341","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-18908][SS] Creating StreamingQueryException should check if logicalPlan is created\n\n## What changes were proposed in this pull request?\n\nThis PR audits places using `logicalPlan` in StreamExecution and ensures they all handles the case that `logicalPlan` cannot be created.\n\nIn addition, this PR also fixes the following issues in `StreamingQueryException`:\n- `StreamingQueryException` and `StreamExecution` are cycle-dependent because in the `StreamingQueryException`'s constructor, it calls `StreamExecution`'s `toDebugString` which uses `StreamingQueryException`. Hence it will output `null` value in the error message.\n- Duplicated stack trace when calling Throwable.printStackTrace because StreamingQueryException's toString contains the stack trace.\n\n## How was this patch tested?\n\nThe updated `test(\"max files per trigger - incorrect values\")`. I found this issue when I switched from `testStream` to the real codes to verify the failure in this test.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #16322 from zsxwing/SPARK-18907.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/ff7d82a207e8bef7779c27378f7a50a138627341"},{"sha":"7c5b7b3a2e5a7c1b2d0d8ce655840cad581e47ac","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SQL] Minor readability improvement for partition handling code\n\n## What changes were proposed in this pull request?\nThis patch includes minor changes to improve readability for partition handling code. I'm in the middle of implementing some new feature and found some naming / implicit type inference not as intuitive.\n\n## How was this patch tested?\nThis patch should have no semantic change and the changes should be covered by existing test cases.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #16378 from rxin/minor-fix.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/7c5b7b3a2e5a7c1b2d0d8ce655840cad581e47ac"},{"sha":"2e861df96eacd821edbbd9883121bff67611074f","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[DOC] bucketing is applicable to all file-based data sources\n\n## What changes were proposed in this pull request?\nStarting Spark 2.1.0, bucketing feature is available for all file-based data sources. This patch fixes some function docs that haven't yet been updated to reflect that.\n\n## How was this patch tested?\nN/A\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #16349 from rxin/ds-doc.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/2e861df96eacd821edbbd9883121bff67611074f"},{"sha":"f489339c7532c3bdf1d95a6ed93347e487f05c50","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-18953][CORE][WEB UI] Do now show the link to a dead worker on the master page\n\n## What changes were proposed in this pull request?\n\nFor a dead worker, we will not be able to see its worker page anyway. This PR removes the links to dead workers from the master page.\n\n## How was this patch tested?\n\nSince this is UI change, please do the following steps manually.\n\n**1. Start a master and a slave**\n\n```\nsbin/start-master.sh\nsbin/start-slave.sh spark://10.22.16.140:7077\n```\n\n![1_live_worker_a](https://cloud.githubusercontent.com/assets/9700541/21373572/d7e187d6-c6d4-11e6-9110-f4371d215dec.png)\n\n**2. Stop the slave**\n```\nsbin/stop-slave.sh\n```\n\n![2_dead_worker_a](https://cloud.githubusercontent.com/assets/9700541/21373579/dd9e9704-c6d4-11e6-9047-a22cb0aa83ed.png)\n\n**3. Start a slave**\n\n```\nsbin/start-slave.sh spark://10.22.16.140:7077\n```\n\n![3_dead_worder_a_and_live_worker_b](https://cloud.githubusercontent.com/assets/9700541/21373582/e1b207f4-c6d4-11e6-89cb-6d8970175a5e.png)\n\n**4. Stop the slave**\n\n```\nsbin/stop-slave.sh\n```\n\n![4_dead_worker_a_and_b](https://cloud.githubusercontent.com/assets/9700541/21373584/e5fecb4e-c6d4-11e6-95d3-49defe366946.png)\n\n**5. Driver list testing**\n\nDo the followings and stop the slave in a minute by `sbin/stop-slave.sh`.\n\n```\nsbin/start-master.sh\nsbin/start-slave.sh spark://10.22.16.140:7077\nbin/spark-submit --master=spark://10.22.16.140:7077 --deploy-mode=cluster --class org.apache.spark.examples.SparkPi examples/target/scala-2.11/jars/spark-examples_2.11-2.2.0-SNAPSHOT.jar 10000\n```\n\n![5_dead_worker_in_driver_list](https://cloud.githubusercontent.com/assets/9700541/21401320/be6cc9fc-c768-11e6-8de7-6512961296a5.png)\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #16366 from dongjoon-hyun/SPARK-18953.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/f489339c7532c3bdf1d95a6ed93347e487f05c50"},{"sha":"4186aba632eaee2cc2c1ba6906449375c98b6c5c","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-18922][TESTS] Fix more resource-closing-related and path-related test failures in identified ones on Windows\n\n## What changes were proposed in this pull request?\n\nThere are several tests failing due to resource-closing-related and path-related  problems on Windows as below.\n\n- `SQLQuerySuite`:\n\n```\n- specifying database name for a temporary table is not allowed *** FAILED *** (125 milliseconds)\n  org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark  arget mpspark-1f4471ab-aac0-4239-ae35-833d54b37e52;\n  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:382)\n  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:370)\n```\n\n- `JsonSuite`:\n\n```\n- Loading a JSON dataset from a text file with SQL *** FAILED *** (94 milliseconds)\n  org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark  arget mpspark-c918a8b7-fc09-433c-b9d0-36c0f78ae918;\n  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:382)\n  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:370)\n```\n\n- `StateStoreSuite`:\n\n```\n- SPARK-18342: commit fails when rename fails *** FAILED *** (16 milliseconds)\n  java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: StateStoreSuite29777261fs://C:%5Cprojects%5Cspark%5Ctarget%5Ctmp%5Cspark-ef349862-7281-4963-aaf3-add0d670a4ad%5C?????-2218c2f8-2cf6-4f80-9cdf-96354e8246a77685899733421033312/0\n  at org.apache.hadoop.fs.Path.initialize(Path.java:206)\n  at org.apache.hadoop.fs.Path.<init>(Path.java:116)\n  at org.apache.hadoop.fs.Path.<init>(Path.java:89)\n  ...\n  Cause: java.net.URISyntaxException: Relative path in absolute URI: StateStoreSuite29777261fs://C:%5Cprojects%5Cspark%5Ctarget%5Ctmp%5Cspark-ef349862-7281-4963-aaf3-add0d670a4ad%5C?????-2218c2f8-2cf6-4f80-9cdf-96354e8246a77685899733421033312/0\n  at java.net.URI.checkPath(URI.java:1823)\n  at java.net.URI.<init>(URI.java:745)\n  at org.apache.hadoop.fs.Path.initialize(Path.java:203)\n```\n\n- `HDFSMetadataLogSuite`:\n\n```\n- FileManager: FileContextManager *** FAILED *** (94 milliseconds)\n  java.io.IOException: Failed to delete: C:\\projects\\spark\\target\\tmp\\spark-415bb0bd-396b-444d-be82-04599e025f21\n  at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1010)\n  at org.apache.spark.sql.test.SQLTestUtils$class.withTempDir(SQLTestUtils.scala:127)\n  at org.apache.spark.sql.execution.streaming.HDFSMetadataLogSuite.withTempDir(HDFSMetadataLogSuite.scala:38)\n\n- FileManager: FileSystemManager *** FAILED *** (78 milliseconds)\n  java.io.IOException: Failed to delete: C:\\projects\\spark\\target\\tmp\\spark-ef8222cd-85aa-47c0-a396-bc7979e15088\n  at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1010)\n  at org.apache.spark.sql.test.SQLTestUtils$class.withTempDir(SQLTestUtils.scala:127)\n  at org.apache.spark.sql.execution.streaming.HDFSMetadataLogSuite.withTempDir(HDFSMetadataLogSuite.scala:38)\n```\n\nAnd, there are some tests being failed due to the length limitation on cmd in Windows as below:\n\n- `LauncherBackendSuite`:\n\n```\n- local: launcher handle *** FAILED *** (30 seconds, 120 milliseconds)\n  The code passed to eventually never returned normally. Attempted 283 times over 30.0960053 seconds. Last failure message: The reference was null. (LauncherBackendSuite.scala:56)\n  org.scalatest.exceptions.TestFailedDueToTimeoutException:\n  at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420)\n  at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)\n\n- standalone/client: launcher handle *** FAILED *** (30 seconds, 47 milliseconds)\n  The code passed to eventually never returned normally. Attempted 282 times over 30.037987100000002 seconds. Last failure message: The reference was null. (LauncherBackendSuite.scala:56)\n  org.scalatest.exceptions.TestFailedDueToTimeoutException:\n  at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420)\n  at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)\n```\n\nThe executed command is, https://gist.github.com/HyukjinKwon/d3fdd2e694e5c022992838a618a516bd, which is 16K length; however, the length limitation is 8K on Windows. So, it is being failed to launch.\n\nThis PR proposes to fix the test failures on Windows and skip the tests failed due to the length limitation\n\n## How was this patch tested?\n\nManually tested via AppVeyor\n\n**Before**\n\n`SQLQuerySuite `: https://ci.appveyor.com/project/spark-test/spark/build/306-pr-references\n`JsonSuite`: https://ci.appveyor.com/project/spark-test/spark/build/307-pr-references\n`StateStoreSuite` : https://ci.appveyor.com/project/spark-test/spark/build/305-pr-references\n`HDFSMetadataLogSuite`: https://ci.appveyor.com/project/spark-test/spark/build/304-pr-references\n`LauncherBackendSuite`: https://ci.appveyor.com/project/spark-test/spark/build/303-pr-references\n\n**After**\n\n`SQLQuerySuite`: https://ci.appveyor.com/project/spark-test/spark/build/293-SQLQuerySuite\n`JsonSuite`: https://ci.appveyor.com/project/spark-test/spark/build/294-JsonSuite\n`StateStoreSuite`: https://ci.appveyor.com/project/spark-test/spark/build/297-StateStoreSuite\n`HDFSMetadataLogSuite`: https://ci.appveyor.com/project/spark-test/spark/build/319-pr-references\n`LauncherBackendSuite`: failed test skipped.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #16335 from HyukjinKwon/more-fixes-on-windows.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/4186aba632eaee2cc2c1ba6906449375c98b6c5c"},{"sha":"76622c661fcae81eb0352c61f54a2e9e21a4fb98","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-16975][SQL][FOLLOWUP] Do not duplicately check file paths in data sources implementing FileFormat\n\n## What changes were proposed in this pull request?\n\nThis PR cleans up duplicated checking for file paths in implemented data sources and prevent to attempt to list twice in ORC data source.\n\nhttps://github.com/apache/spark/pull/14585 handles a problem for the partition column name having `_` and the issue itself is resolved correctly. However, it seems the data sources implementing `FileFormat` are validating the paths duplicately. Assuming from the comment in `CSVFileFormat`, `// TODO: Move filtering.`, I guess we don't have to check this duplicately.\n\n   Currently, this seems being filtered in `PartitioningAwareFileIndex.shouldFilterOut` and`PartitioningAwareFileIndex.isDataPath`. So, `FileFormat.inferSchema` will always receive leaf files. For example, running to codes below:\n\n   ``` scala\n   spark.range(10).withColumn(\"_locality_code\", $\"id\").write.partitionBy(\"_locality_code\").save(\"/tmp/parquet\")\n   spark.read.parquet(\"/tmp/parquet\")\n   ```\n\n   gives the paths below without directories but just valid data files:\n\n   ``` bash\n   /tmp/parquet/_col=0/part-r-00000-094a8efa-bece-4b50-b54c-7918d1f7b3f8.snappy.parquet\n   /tmp/parquet/_col=1/part-r-00000-094a8efa-bece-4b50-b54c-7918d1f7b3f8.snappy.parquet\n   /tmp/parquet/_col=2/part-r-00000-25de2b50-225a-4bcf-a2bc-9eb9ed407ef6.snappy.parquet\n   ...\n   ```\n\n   to `FileFormat.inferSchema`.\n\n## How was this patch tested?\n\nUnit test added in `HadoopFsRelationTest` and related existing tests.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14627 from HyukjinKwon/SPARK-16975.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/76622c661fcae81eb0352c61f54a2e9e21a4fb98"},{"sha":"2615100055860faa5f74d3711d4d15ebae6aba25","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-18973][SQL] Remove SortPartitions and RedistributeData\n\n## What changes were proposed in this pull request?\nSortPartitions and RedistributeData logical operators are not actually used and can be removed. Note that we do have a Sort operator (with global flag false) that subsumed SortPartitions.\n\n## How was this patch tested?\nAlso updated test cases to reflect the removal.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #16381 from rxin/SPARK-18973.","distinct":true,"url":"https://api.github.com/repos/darionyaphet/spark/commits/2615100055860faa5f74d3711d4d15ebae6aba25"}]}
{"actor":{"display_login":"darionyaphet"}}