{"push_id":1642358425,"size":173,"distinct_size":173,"ref":"refs/heads/spark-19868","head":"fd6739255df23c54e7c19080d73eeeca389754b4","before":"17acd55f57498195360d4b5876047c92005c3a68","commits":[{"sha":"3f9f9180c2e695ad468eb813df5feec41e169531","author":{"name":"Yuming Wang","email":"b4045a50547081d83941485c6596bd2032665522@gmail.com"},"message":"[SPARK-19693][SQL] Make the SET mapreduce.job.reduces automatically converted to spark.sql.shuffle.partitions\n\n## What changes were proposed in this pull request?\nMake the `SET mapreduce.job.reduces` automatically converted to `spark.sql.shuffle.partitions`, it's similar to `SET mapred.reduce.tasks`.\n\n## How was this patch tested?\n\nunit tests\n\nAuthor: Yuming Wang <wgyumg@gmail.com>\n\nCloses #17020 from wangyum/SPARK-19693.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/3f9f9180c2e695ad468eb813df5feec41e169531"},{"sha":"9ea201cf6482c9c62c9428759d238063db62d66e","author":{"name":"Anthony Truchet","email":"26d8dd5b3efdf5761f8c8003329681520fd47d3c@criteo.com"},"message":"[SPARK-16440][MLLIB] Ensure broadcasted variables are destroyed even in case of exception\n\n## What changes were proposed in this pull request?\n\nEnsure broadcasted variable are destroyed even in case of exception\n## How was this patch tested?\n\nWord2VecSuite was run locally\n\nAuthor: Anthony Truchet <a.truchet@criteo.com>\n\nCloses #14299 from AnthonyTruchet/SPARK-16440.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/9ea201cf6482c9c62c9428759d238063db62d66e"},{"sha":"e44274870dee308f4e3e8ce79457d8d19693b6e5","author":{"name":"wangzhenhua","email":"93943f521ced4ea3f21d64a737435e7a7e171ac6@huawei.com"},"message":"[SPARK-17080][SQL] join reorder\n\n## What changes were proposed in this pull request?\n\nReorder the joins using a dynamic programming algorithm (Selinger paper):\nFirst we put all items (basic joined nodes) into level 1, then we build all two-way joins at level 2 from plans at level 1 (single items), then build all 3-way joins from plans at previous levels (two-way joins and single items), then 4-way joins ... etc, until we build all n-way joins and pick the best plan among them.\n\nWhen building m-way joins, we only keep the best plan (with the lowest cost) for the same set of m items. E.g., for 3-way joins, we keep only the best plan for items {A, B, C} among plans (A J B) J C, (A J C) J B and (B J C) J A. Thus, the plans maintained for each level when reordering four items A, B, C, D are as follows:\n```\nlevel 1: p({A}), p({B}), p({C}), p({D})\nlevel 2: p({A, B}), p({A, C}), p({A, D}), p({B, C}), p({B, D}), p({C, D})\nlevel 3: p({A, B, C}), p({A, B, D}), p({A, C, D}), p({B, C, D})\nlevel 4: p({A, B, C, D})\n```\nwhere p({A, B, C, D}) is the final output plan.\n\nFor cost evaluation, since physical costs for operators are not available currently, we use cardinalities and sizes to compute costs.\n\n## How was this patch tested?\nadd test cases\n\nAuthor: wangzhenhua <wangzhenhua@huawei.com>\nAuthor: Zhenhua Wang <wzh_zju@163.com>\n\nCloses #17138 from wzhfy/joinReorder.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/e44274870dee308f4e3e8ce79457d8d19693b6e5"},{"sha":"5f7d835d380c1a558a4a6d8366140cd96ee202eb","author":{"name":"jiangxingbo","email":"5b9d05a325a3f85d4c824395342a210596784248@gmail.com"},"message":"[SPARK-19865][SQL] remove the view identifier in SubqueryAlias\n\n## What changes were proposed in this pull request?\n\nSince we have a `View` node now, we can remove the view identifier in `SubqueryAlias`, which was used to indicate a view node before.\n\n## How was this patch tested?\n\nUpdate the related test cases.\n\nAuthor: jiangxingbo <jiangxb1987@gmail.com>\n\nCloses #17210 from jiangxb1987/SubqueryAlias.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/5f7d835d380c1a558a4a6d8366140cd96ee202eb"},{"sha":"9a6ac7226fd09d570cae08d0daea82d9bca189a0","author":{"name":"Xiao Li","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-19601][SQL] Fix CollapseRepartition rule to preserve shuffle-enabled Repartition\n\n### What changes were proposed in this pull request?\n\nObserved by felixcheung  in https://github.com/apache/spark/pull/16739, when users use the shuffle-enabled `repartition` API, they expect the partition they got should be the exact number they provided, even if they call shuffle-disabled `coalesce` later.\n\nCurrently, `CollapseRepartition` rule does not consider whether shuffle is enabled or not. Thus, we got the following unexpected result.\n\n```Scala\n    val df = spark.range(0, 10000, 1, 5)\n    val df2 = df.repartition(10)\n    assert(df2.coalesce(13).rdd.getNumPartitions == 5)\n    assert(df2.coalesce(7).rdd.getNumPartitions == 5)\n    assert(df2.coalesce(3).rdd.getNumPartitions == 3)\n```\n\nThis PR is to fix the issue. We preserve shuffle-enabled Repartition.\n\n### How was this patch tested?\nAdded a test case\n\nAuthor: Xiao Li <gatorsmile@gmail.com>\n\nCloses #16933 from gatorsmile/CollapseRepartition.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/9a6ac7226fd09d570cae08d0daea82d9bca189a0"},{"sha":"e420fd4592615d91cdcbca674ac58bcca6ab2ff3","author":{"name":"Tejas Patil","email":"6ad8f8971953bfaf2a65cdbf75ed01ef8f5bdc87@fb.com"},"message":"[SPARK-19843][SQL][FOLLOWUP] Classdoc for `IntWrapper` and `LongWrapper`\n\n## What changes were proposed in this pull request?\n\nThis is as per suggestion by rxin at : https://github.com/apache/spark/pull/17184#discussion_r104841735\n\n## How was this patch tested?\n\nNA as this is a documentation change\n\nAuthor: Tejas Patil <tejasp@fb.com>\n\nCloses #17205 from tejasapatil/SPARK-19843_followup.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/e420fd4592615d91cdcbca674ac58bcca6ab2ff3"},{"sha":"f3387d97487cbef894b6963bc008f6a5c4294a85","author":{"name":"windpiger","email":"83bba5ef3ac29b005cb86c055a312c08a5e441d9@outlook.com"},"message":"[SPARK-19864][SQL][TEST] provide a makeQualifiedPath functions to optimize some code\n\n## What changes were proposed in this pull request?\n\nCurrently there are lots of places to make the path qualified, it is better to provide a function to do this, then the code will be more simple.\n\n## How was this patch tested?\nN/A\n\nAuthor: windpiger <songjun@outlook.com>\n\nCloses #17204 from windpiger/addQualifiledPathUtil.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/f3387d97487cbef894b6963bc008f6a5c4294a85"},{"sha":"e9e2c612d58a19ddcb4b6abfb7389a4b0f7ef6f8","author":{"name":"Wojtek Szymanski","email":"bee123fce4957f4bf409d8345fc1bc7fed03dd55@gmail.com"},"message":"[SPARK-19727][SQL] Fix for round function that modifies original column\n\n## What changes were proposed in this pull request?\n\nFix for SQL round function that modifies original column when underlying data frame is created from a local product.\n\n    import org.apache.spark.sql.functions._\n\n    case class NumericRow(value: BigDecimal)\n\n    val df = spark.createDataFrame(Seq(NumericRow(BigDecimal(\"1.23456789\"))))\n\n    df.show()\n    +--------------------+\n    |               value|\n    +--------------------+\n    |1.234567890000000000|\n    +--------------------+\n\n    df.withColumn(\"value_rounded\", round('value)).show()\n\n    // before\n    +--------------------+-------------+\n    |               value|value_rounded|\n    +--------------------+-------------+\n    |1.000000000000000000|            1|\n    +--------------------+-------------+\n\n    // after\n    +--------------------+-------------+\n    |               value|value_rounded|\n    +--------------------+-------------+\n    |1.234567890000000000|            1|\n    +--------------------+-------------+\n\n## How was this patch tested?\n\nNew unit test added to existing suite `org.apache.spark.sql.MathFunctionsSuite`\n\nAuthor: Wojtek Szymanski <wk.szymanski@gmail.com>\n\nCloses #17075 from wojtek-szymanski/SPARK-19727.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/e9e2c612d58a19ddcb4b6abfb7389a4b0f7ef6f8"},{"sha":"1bf9012380de2aa7bdf39220b55748defde8b700","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-19858][SS] Add output mode to flatMapGroupsWithState and disallow invalid cases\n\n## What changes were proposed in this pull request?\n\nAdd a output mode parameter to `flatMapGroupsWithState` and just define `mapGroupsWithState` as `flatMapGroupsWithState(Update)`.\n\n`UnsupportedOperationChecker` is modified to disallow unsupported cases.\n\n- Batch mapGroupsWithState or flatMapGroupsWithState is always allowed.\n- For streaming (map/flatMap)GroupsWithState, see the following table:\n\n| Operators  | Supported Query Output Mode |\n| ------------- | ------------- |\n| flatMapGroupsWithState(Update) without aggregation  | Update |\n| flatMapGroupsWithState(Update) with aggregation  | None |\n| flatMapGroupsWithState(Append) without aggregation  | Append |\n| flatMapGroupsWithState(Append) before aggregation  | Append, Update, Complete |\n| flatMapGroupsWithState(Append) after aggregation  | None |\n| Multiple flatMapGroupsWithState(Append)s  | Append |\n| Multiple mapGroupsWithStates  | None |\n| Mxing mapGroupsWithStates  and flatMapGroupsWithStates | None |\n| Other cases of multiple flatMapGroupsWithState | None |\n\n## How was this patch tested?\n\nThe added unit tests. Here are the tests related to (map/flatMap)GroupsWithState:\n```\n[info] - batch plan - flatMapGroupsWithState - flatMapGroupsWithState(Append) on batch relation: supported (1 millisecond)\n[info] - batch plan - flatMapGroupsWithState - multiple flatMapGroupsWithState(Append)s on batch relation: supported (0 milliseconds)\n[info] - batch plan - flatMapGroupsWithState - flatMapGroupsWithState(Update) on batch relation: supported (0 milliseconds)\n[info] - batch plan - flatMapGroupsWithState - multiple flatMapGroupsWithState(Update)s on batch relation: supported (0 milliseconds)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Update) on streaming relation without aggregation in update mode: supported (2 milliseconds)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Update) on streaming relation without aggregation in append mode: not supported (7 milliseconds)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Update) on streaming relation without aggregation in complete mode: not supported (5 milliseconds)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Update) on streaming relation with aggregation in Append mode: not supported (11 milliseconds)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Update) on streaming relation with aggregation in Update mode: not supported (5 milliseconds)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Update) on streaming relation with aggregation in Complete mode: not supported (5 milliseconds)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Append) on streaming relation without aggregation in append mode: supported (1 millisecond)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Append) on streaming relation without aggregation in update mode: not supported (6 milliseconds)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Append) on streaming relation before aggregation in Append mode: supported (1 millisecond)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Append) on streaming relation before aggregation in Update mode: supported (0 milliseconds)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Append) on streaming relation before aggregation in Complete mode: supported (1 millisecond)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Append) on streaming relation after aggregation in Append mode: not supported (6 milliseconds)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Append) on streaming relation after aggregation in Update mode: not supported (4 milliseconds)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Update) on streaming relation in complete mode: not supported (2 milliseconds)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Append) on batch relation inside streaming relation in Append output mode: supported (1 millisecond)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Append) on batch relation inside streaming relation in Update output mode: supported (1 millisecond)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Update) on batch relation inside streaming relation in Append output mode: supported (0 milliseconds)\n[info] - streaming plan - flatMapGroupsWithState - flatMapGroupsWithState(Update) on batch relation inside streaming relation in Update output mode: supported (0 milliseconds)\n[info] - streaming plan - flatMapGroupsWithState - multiple flatMapGroupsWithStates on streaming relation and all are in append mode: supported (2 milliseconds)\n[info] - streaming plan - flatMapGroupsWithState -  multiple flatMapGroupsWithStates on s streaming relation but some are not in append mode: not supported (7 milliseconds)\n[info] - streaming plan - mapGroupsWithState - mapGroupsWithState on streaming relation without aggregation in append mode: not supported (3 milliseconds)\n[info] - streaming plan - mapGroupsWithState - mapGroupsWithState on streaming relation without aggregation in complete mode: not supported (3 milliseconds)\n[info] - streaming plan - mapGroupsWithState - mapGroupsWithState on streaming relation with aggregation in Append mode: not supported (6 milliseconds)\n[info] - streaming plan - mapGroupsWithState - mapGroupsWithState on streaming relation with aggregation in Update mode: not supported (3 milliseconds)\n[info] - streaming plan - mapGroupsWithState - mapGroupsWithState on streaming relation with aggregation in Complete mode: not supported (4 milliseconds)\n[info] - streaming plan - mapGroupsWithState - multiple mapGroupsWithStates on streaming relation and all are in append mode: not supported (4 milliseconds)\n[info] - streaming plan - mapGroupsWithState - mixing mapGroupsWithStates and flatMapGroupsWithStates on streaming relation: not supported (4 milliseconds)\n```\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #17197 from zsxwing/mapgroups-check.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/1bf9012380de2aa7bdf39220b55748defde8b700"},{"sha":"6570cfd7abe349dc6d2151f2ac9dc662e7465a79","author":{"name":"Kunal Khamar","email":"224bcdd195840375ef11e587bc3c973befb5ce34@outlook.com"},"message":"[SPARK-19540][SQL] Add ability to clone SparkSession wherein cloned session has an identical copy of the SessionState\n\nForking a newSession() from SparkSession currently makes a new SparkSession that does not retain SessionState (i.e. temporary tables, SQL config, registered functions etc.) This change adds a method cloneSession() which creates a new SparkSession with a copy of the parent's SessionState.\n\nSubsequent changes to base session are not propagated to cloned session, clone is independent after creation.\nIf the base is changed after clone has been created, say user registers new UDF, then the new UDF will not be available inside the clone. Same goes for configs and temp tables.\n\nUnit tests\n\nAuthor: Kunal Khamar <kkhamar@outlook.com>\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #16826 from kunalkhamar/fork-sparksession.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/6570cfd7abe349dc6d2151f2ac9dc662e7465a79"},{"sha":"455129020ca7f6a162f6f2486a87cc43512cfd2c","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-15463][SQL] Add an API to load DataFrame from Dataset[String] storing CSV\n\n## What changes were proposed in this pull request?\n\nThis PR proposes to add an API that loads `DataFrame` from `Dataset[String]` storing csv.\n\nIt allows pre-processing before loading into CSV, which means allowing a lot of workarounds for many narrow cases, for example, as below:\n\n- Case 1 - pre-processing\n\n  ```scala\n  val df = spark.read.text(\"...\")\n  // Pre-processing with this.\n  spark.read.csv(df.as[String])\n  ```\n\n- Case 2 - use other input formats\n\n  ```scala\n  val rdd = spark.sparkContext.newAPIHadoopFile(\"/file.csv.lzo\",\n    classOf[com.hadoop.mapreduce.LzoTextInputFormat],\n    classOf[org.apache.hadoop.io.LongWritable],\n    classOf[org.apache.hadoop.io.Text])\n  val stringRdd = rdd.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength))\n\n  spark.read.csv(stringRdd.toDS)\n  ```\n\n## How was this patch tested?\n\nAdded tests in `CSVSuite` and build with Scala 2.10.\n\n```\n./dev/change-scala-version.sh 2.10\n./build/mvn -Pyarn -Phadoop-2.4 -Dscala-2.10 -DskipTests clean package\n```\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #16854 from HyukjinKwon/SPARK-15463.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/455129020ca7f6a162f6f2486a87cc43512cfd2c"},{"sha":"a3648b5d4f99ff9461d02f53e9ec71787a3abf51","author":{"name":"Burak Yavuz","email":"39bd9266538731f16447e7d968c0ba25275ede1e@gmail.com"},"message":"[SPARK-19813] maxFilesPerTrigger combo latestFirst may miss old files in combination with maxFileAge in FileStreamSource\n\n## What changes were proposed in this pull request?\n\n**The Problem**\nThere is a file stream source option called maxFileAge which limits how old the files can be, relative the latest file that has been seen. This is used to limit the files that need to be remembered as \"processed\". Files older than the latest processed files are ignored. This values is by default 7 days.\nThis causes a problem when both\nlatestFirst = true\nmaxFilesPerTrigger > total files to be processed.\nHere is what happens in all combinations\n1) latestFirst = false - Since files are processed in order, there wont be any unprocessed file older than the latest processed file. All files will be processed.\n2) latestFirst = true AND maxFilesPerTrigger is not set - The maxFileAge thresholding mechanism takes one batch initialize. If maxFilesPerTrigger is not, then all old files get processed in the first batch, and so no file is left behind.\n3) latestFirst = true AND maxFilesPerTrigger is set to X - The first batch process the latest X files. That sets the threshold latest file - maxFileAge, so files older than this threshold will never be considered for processing.\nThe bug is with case 3.\n\n**The Solution**\n\nIgnore `maxFileAge` when both `maxFilesPerTrigger` and `latestFirst` are set.\n\n## How was this patch tested?\n\nRegression test in `FileStreamSourceSuite`\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #17153 from brkyvz/maxFileAge.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/a3648b5d4f99ff9461d02f53e9ec71787a3abf51"},{"sha":"d809ceed9762d5bbb04170e45f38751713112dd8","author":{"name":"Dilip Biswal","email":"6ab79ed79ae7ed4bf1dfe07d1c3e4c433e1f76a0@us.ibm.com"},"message":"[MINOR][SQL] The analyzer rules are fired twice for cases when AnalysisException is raised from analyzer.\n\n## What changes were proposed in this pull request?\nIn general we have a checkAnalysis phase which validates the logical plan and throws AnalysisException on semantic errors. However we also can throw AnalysisException from a few analyzer rules like ResolveSubquery.\n\nI found that we fire up the analyzer rules twice for the queries that throw AnalysisException from one of the analyzer rules. This is a very minor fix. We don't have to strictly fix it. I just got confused seeing the rule getting fired two times when i was not expecting it.\n\n## How was this patch tested?\n\nTested manually.\n\nAuthor: Dilip Biswal <dbiswal@us.ibm.com>\n\nCloses #17214 from dilipbiswal/analyis_twice.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/d809ceed9762d5bbb04170e45f38751713112dd8"},{"sha":"09829be621f0f9bb5076abb3d832925624699fa9","author":{"name":"Xiao Li","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-19235][SQL][TESTS] Enable Test Cases in DDLSuite with Hive Metastore\n\n### What changes were proposed in this pull request?\nSo far, the test cases in DDLSuites only verify the behaviors of InMemoryCatalog. That means, they do not cover the scenarios using HiveExternalCatalog. Thus, we need to improve the existing test suite to run these cases using Hive metastore.\n\nWhen porting these test cases, a bug of `SET LOCATION` is found. `path` is not set when the location is changed.\n\nAfter this PR, a few changes are made, as summarized below,\n- `DDLSuite` becomes an abstract class. Both `InMemoryCatalogedDDLSuite` and `HiveCatalogedDDLSuite` extend it. `InMemoryCatalogedDDLSuite` is using `InMemoryCatalog`. `HiveCatalogedDDLSuite` is using `HiveExternalCatalog`.\n- `InMemoryCatalogedDDLSuite` contains all the existing test cases in `DDLSuite`.\n- `HiveCatalogedDDLSuite` contains a subset of `DDLSuite`. The following test cases are excluded:\n\n1. The following test cases only make sense for `InMemoryCatalog`:\n```\n  test(\"desc table for parquet data source table using in-memory catalog\")\n  test(\"create a managed Hive source table\") {\n  test(\"create an external Hive source table\")\n  test(\"Create Hive Table As Select\")\n```\n\n2. The following test cases are unable to be ported because we are unable to alter table provider when using Hive metastore. In the future PRs we need to improve the test cases so that altering table provider is not needed:\n```\n  test(\"alter table: set location (datasource table)\")\n  test(\"alter table: set properties (datasource table)\")\n  test(\"alter table: unset properties (datasource table)\")\n  test(\"alter table: set serde (datasource table)\")\n  test(\"alter table: set serde partition (datasource table)\")\n  test(\"alter table: change column (datasource table)\")\n  test(\"alter table: add partition (datasource table)\")\n  test(\"alter table: drop partition (datasource table)\")\n  test(\"alter table: rename partition (datasource table)\")\n  test(\"drop table - data source table\")\n```\n\n**TODO** : in the future PRs, we need to remove `HiveDDLSuite` and move the test cases to either `DDLSuite`,  `InMemoryCatalogedDDLSuite` or `HiveCatalogedDDLSuite`.\n\n### How was this patch tested?\nN/A\n\nAuthor: Xiao Li <gatorsmile@gmail.com>\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #16592 from gatorsmile/refactorDDLSuite.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/09829be621f0f9bb5076abb3d832925624699fa9"},{"sha":"029e40b412e332c9f0fff283d604e203066c78c0","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-19874][BUILD] Hide API docs for org.apache.spark.sql.internal\n\n## What changes were proposed in this pull request?\n\nThe API docs should not include the \"org.apache.spark.sql.internal\" package because they are internal private APIs.\n\n## How was this patch tested?\n\nJenkins\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #17217 from zsxwing/SPARK-19874.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/029e40b412e332c9f0fff283d604e203066c78c0"},{"sha":"eeb1d6db878641d9eac62d0869a90fe80c1f4461","author":{"name":"uncleGen","email":"6269a8ea04865dd92a1986593446a029690f118a@gmail.com"},"message":"[SPARK-19859][SS][FOLLOW-UP] The new watermark should override the old one.\n\n## What changes were proposed in this pull request?\n\nA follow up to SPARK-19859:\n\n- extract the calculation of `delayMs` and reuse it.\n- update EventTimeWatermarkExec\n- use the correct `delayMs` in EventTimeWatermark\n\n## How was this patch tested?\n\nJenkins.\n\nAuthor: uncleGen <hustyugm@gmail.com>\n\nCloses #17221 from uncleGen/SPARK-19859.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/eeb1d6db878641d9eac62d0869a90fe80c1f4461"},{"sha":"274973d2a32ff4eb28545b50a3135e4784eb3047","author":{"name":"windpiger","email":"83bba5ef3ac29b005cb86c055a312c08a5e441d9@outlook.com"},"message":"[SPARK-19763][SQL] qualified external datasource table location stored in catalog\n\n## What changes were proposed in this pull request?\n\nIf we create a external datasource table with a non-qualified location , we should qualified it to store in catalog.\n\n```\nCREATE TABLE t(a string)\nUSING parquet\nLOCATION '/path/xx'\n\nCREATE TABLE t1(a string, b string)\nUSING parquet\nPARTITIONED BY(b)\nLOCATION '/path/xx'\n```\n\nwhen we get the table from catalog, the location should be qualified, e.g.'file:/path/xxx'\n## How was this patch tested?\nunit test added\n\nAuthor: windpiger <songjun@outlook.com>\n\nCloses #17095 from windpiger/tablepathQualified.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/274973d2a32ff4eb28545b50a3135e4784eb3047"},{"sha":"206030bd12405623c00c1ff334663984b9250adb","author":{"name":"Jason White","email":"9058cd9d5bfe65be4f261e7ff83f4a2726aac0ba@shopify.com"},"message":"[SPARK-19561][SQL] add int case handling for TimestampType\n\n## What changes were proposed in this pull request?\n\nAdd handling of input of type `Int` for dataType `TimestampType` to `EvaluatePython.scala`. Py4J serializes ints smaller than MIN_INT or larger than MAX_INT to Long, which are handled correctly already, but values between MIN_INT and MAX_INT are serialized to Int.\n\nThese range limits correspond to roughly half an hour on either side of the epoch. As a result, PySpark doesn't allow TimestampType values to be created in this range.\n\nAlternatives attempted: patching the `TimestampType.toInternal` function to cast return values to `long`, so Py4J would always serialize them to Scala Long. Python3 does not have a `long` type, so this approach failed on Python3.\n\n## How was this patch tested?\n\nAdded a new PySpark-side test that fails without the change.\n\nThe contribution is my original work and I license the work to the project under the project’s open source license.\n\nResubmission of https://github.com/apache/spark/pull/16896. The original PR didn't go through Jenkins and broke the build. davies dongjoon-hyun\n\ncloud-fan Could you kick off a Jenkins run for me? It passed everything for me locally, but it's possible something has changed in the last few weeks.\n\nAuthor: Jason White <jason.white@shopify.com>\n\nCloses #17200 from JasonMWhite/SPARK-19561.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/206030bd12405623c00c1ff334663984b9250adb"},{"sha":"b60b9fc10a1ee52c3c021a4a5faf10f92f83e3c9","author":{"name":"Jimmy Xiang","email":"5f517ef57cc8e3a03baf25802f43d194442b2595@apache.org"},"message":"[SPARK-19757][CORE] DriverEndpoint#makeOffers race against CoarseGrainedSchedulerBackend#killExecutors\n\n## What changes were proposed in this pull request?\nWhile some executors are being killed due to idleness, if some new tasks come in, driver could assign them to some executors are being killed. These tasks will fail later when the executors are lost. This patch is to make sure CoarseGrainedSchedulerBackend#killExecutors and DriverEndpoint#makeOffers are properly synchronized.\n\n## How was this patch tested?\nmanual tests\n\nAuthor: Jimmy Xiang <jxiang@apache.org>\n\nCloses #17091 from jxiang/spark-19757.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/b60b9fc10a1ee52c3c021a4a5faf10f92f83e3c9"},{"sha":"3232e54f2fcb8d2072cba4bc763ef29d5d8d325f","author":{"name":"jinxing","email":"f17febf9b8eb64f774700c766901adf34b63e5a2@126.com"},"message":"[SPARK-19793] Use clock.getTimeMillis when mark task as finished in TaskSetManager.\n\n## What changes were proposed in this pull request?\n\nTaskSetManager is now using `System.getCurrentTimeMillis` when mark task as finished in `handleSuccessfulTask` and `handleFailedTask`. Thus developer cannot set the tasks finishing time in unit test. When `handleSuccessfulTask`, task's duration = `System.getCurrentTimeMillis` - launchTime(which can be set by `clock`), the result is not correct.\n\n## How was this patch tested?\nExisting tests.\n\nAuthor: jinxing <jinxing6042@126.com>\n\nCloses #17133 from jinxing64/SPARK-19793.","distinct":true,"url":"https://api.github.com/repos/liujianhuiouc/spark-1/commits/3232e54f2fcb8d2072cba4bc763ef29d5d8d325f"}]}
{"actor":{"display_login":"liujianhuiouc"}}