{"push_id":1718625815,"size":34,"distinct_size":13,"ref":"refs/heads/SPARK-20550","head":"60fcd8aa8e136b245f9923ca36c68da1338990c9","before":"e06544fc4e0e7c2c430fa7333af1ddd86252c270","commits":[{"sha":"2b2dd08e975dd7fbf261436aa877f1d7497ed31f","author":{"name":"Ryan Blue","email":"6495babc1655ac8354538a68e2908898c1d5fcdc@apache.org"},"message":"[SPARK-20540][CORE] Fix unstable executor requests.\n\nThere are two problems fixed in this commit. First, the\nExecutorAllocationManager sets a timeout to avoid requesting executors\ntoo often. However, the timeout is always updated based on its value and\na timeout, not the current time. If the call is delayed by locking for\nmore than the ongoing scheduler timeout, the manager will request more\nexecutors on every run. This seems to be the main cause of SPARK-20540.\n\nThe second problem is that the total number of requested executors is\nnot tracked by the CoarseGrainedSchedulerBackend. Instead, it calculates\nthe value based on the current status of 3 variables: the number of\nknown executors, the number of executors that have been killed, and the\nnumber of pending executors. But, the number of pending executors is\nnever less than 0, even though there may be more known than requested.\nWhen executors are killed and not replaced, this can cause the request\nsent to YARN to be incorrect because there were too many executors due\nto the scheduler's state being slightly out of date. This is fixed by tracking\nthe currently requested size explicitly.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Ryan Blue <blue@apache.org>\n\nCloses #17813 from rdblue/SPARK-20540-fix-dynamic-allocation.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/2b2dd08e975dd7fbf261436aa877f1d7497ed31f"},{"sha":"af726cd6117de05c6e3b9616b8699d884a53651b","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-20459][SQL] JdbcUtils throws IllegalStateException: Cause already initialized after getting SQLException\n\n## What changes were proposed in this pull request?\n\nAvoid failing to initCause on JDBC exception with cause initialized to null\n\n## How was this patch tested?\n\nExisting tests\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #17800 from srowen/SPARK-20459.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/af726cd6117de05c6e3b9616b8699d884a53651b"},{"sha":"259860d23d1740954b739b639c5bdc3ede65ed25","author":{"name":"ptkool","email":"5f86158097a5e69bbf1a87e339abd8bd5909c221@shopify.com"},"message":"[SPARK-20463] Add support for IS [NOT] DISTINCT FROM.\n\n## What changes were proposed in this pull request?\n\nAdd support for the SQL standard distinct predicate to SPARK SQL.\n\n```\n<expression> IS [NOT] DISTINCT FROM <expression>\n```\n\n## How was this patch tested?\n\nTested using unit tests, integration tests, manual tests.\n\nAuthor: ptkool <michael.styles@shopify.com>\n\nCloses #17764 from ptkool/is_not_distinct_from.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/259860d23d1740954b739b639c5bdc3ede65ed25"},{"sha":"943a684b9827ca294ed06a46431507538d40a134","author":{"name":"Sameer Agarwal","email":"a70383dff9f522db5c95b6d04d9bfe45099f5e1d@cs.berkeley.edu"},"message":"[SPARK-20548] Disable ReplSuite.newProductSeqEncoder with REPL defined class\n\n## What changes were proposed in this pull request?\n\n`newProductSeqEncoder with REPL defined class` in `ReplSuite` has been failing in-deterministically : https://spark-tests.appspot.com/failed-tests over the last few days. Disabling the test until a fix is in place.\n\nhttps://spark.test.databricks.com/job/spark-master-test-sbt-hadoop-2.7/176/testReport/junit/org.apache.spark.repl/ReplSuite/newProductSeqEncoder_with_REPL_defined_class/history/\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Sameer Agarwal <sameerag@cs.berkeley.edu>\n\nCloses #17823 from sameeragarwal/disable-test.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/943a684b9827ca294ed06a46431507538d40a134"},{"sha":"d20a976e8918ca8d607af452301e8014fe14e64a","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-20192][SPARKR][DOC] SparkR migration guide to 2.2.0\n\n## What changes were proposed in this pull request?\n\nUpdating R Programming Guide\n\n## How was this patch tested?\n\nmanually\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #17816 from felixcheung/r22relnote.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/d20a976e8918ca8d607af452301e8014fe14e64a"},{"sha":"90d77e971f6b3fa268e411279f34bc1db4321991","author":{"name":"zero323","email":"173f3ab6399f1b748c2cb687d75e045296a62aa6@users.noreply.github.com"},"message":"[SPARK-20532][SPARKR] Implement grouping and grouping_id\n\n## What changes were proposed in this pull request?\n\nAdds R wrappers for:\n\n- `o.a.s.sql.functions.grouping` as `o.a.s.sql.functions.is_grouping` (to avoid shading `base::grouping`\n- `o.a.s.sql.functions.grouping_id`\n\n## How was this patch tested?\n\nExisting unit tests, additional unit tests. `check-cran.sh`.\n\nAuthor: zero323 <zero323@users.noreply.github.com>\n\nCloses #17807 from zero323/SPARK-20532.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/90d77e971f6b3fa268e411279f34bc1db4321991"},{"sha":"afb21bf22a59c9416c04637412fb69d1442e6826","author":{"name":"Kazuaki Ishizaki","email":"64981c6bd46971e76f308b8a384367244e8169bf@jp.ibm.com"},"message":"[SPARK-20537][CORE] Fixing OffHeapColumnVector reallocation\n\n## What changes were proposed in this pull request?\n\nAs #17773 revealed `OnHeapColumnVector` may copy a part of the original storage.\n\n`OffHeapColumnVector` reallocation also copies to the new storage data up to 'elementsAppended'. This variable is only updated when using the `ColumnVector.appendX` API, while `ColumnVector.putX` is more commonly used.\nThis PR copies the new storage data up to the previously-allocated size in`OffHeapColumnVector`.\n\n## How was this patch tested?\n\nExisting test suites\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #17811 from kiszk/SPARK-20537.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/afb21bf22a59c9416c04637412fb69d1442e6826"},{"sha":"86174ea89b39a300caaba6baffac70f3dc702788","author":{"name":"Burak Yavuz","email":"39bd9266538731f16447e7d968c0ba25275ede1e@gmail.com"},"message":"[SPARK-20549] java.io.CharConversionException: Invalid UTF-32' in JsonToStructs\n\n## What changes were proposed in this pull request?\n\nA fix for the same problem was made in #17693 but ignored `JsonToStructs`. This PR uses the same fix for `JsonToStructs`.\n\n## How was this patch tested?\n\nRegression test\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #17826 from brkyvz/SPARK-20549.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/86174ea89b39a300caaba6baffac70f3dc702788"},{"sha":"e300a5a145820ecd466885c73245d6684e8cb0aa","author":{"name":"Nick Pentreath","email":"01c7185e211fff3925ce587674cd3ea63724775a@za.ibm.com"},"message":"[SPARK-20300][ML][PYSPARK] Python API for ALSModel.recommendForAllUsers,Items\n\nAdd Python API for `ALSModel` methods `recommendForAllUsers`, `recommendForAllItems`\n\n## How was this patch tested?\n\nNew doc tests.\n\nAuthor: Nick Pentreath <nickp@za.ibm.com>\n\nCloses #17622 from MLnick/SPARK-20300-pyspark-recall.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/e300a5a145820ecd466885c73245d6684e8cb0aa"},{"sha":"b1e639ab09d3a7a1545119e45a505c9a04308353","author":{"name":"Xiao Li","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-19235][SQL][TEST][FOLLOW-UP] Enable Test Cases in DDLSuite with Hive Metastore\n\n### What changes were proposed in this pull request?\nThis is a follow-up of enabling test cases in DDLSuite with Hive Metastore. It consists of the following remaining tasks:\n- Run all the `alter table` and `drop table` DDL tests against data source tables when using Hive metastore.\n- Do not run any `alter table` and `drop table` DDL test against Hive serde tables when using InMemoryCatalog.\n- Reenable `alter table: set serde partition` and `alter table: set serde` tests for Hive serde tables.\n\n### How was this patch tested?\nN/A\n\nAuthor: Xiao Li <gatorsmile@gmail.com>\n\nCloses #17524 from gatorsmile/cleanupDDLSuite.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/b1e639ab09d3a7a1545119e45a505c9a04308353"},{"sha":"13f47dc5033a99df8d9ec18f2ce373119462f7bc","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-20490][SPARKR][DOC] add family tag for not function\n\n## What changes were proposed in this pull request?\n\ndoc only\n\n## How was this patch tested?\n\nmanual\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #17828 from felixcheung/rnotfamily.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/13f47dc5033a99df8d9ec18f2ce373119462f7bc"},{"sha":"ef3df9125a30f8fb817fe855b74d7130be45b0ee","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-20421][CORE] Add a missing deprecation tag.\n\nIn the previous patch I deprecated StorageStatus, but not the\nmethod in SparkContext that exposes that class publicly. So deprecate\nthe method too.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #17824 from vanzin/SPARK-20421.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/ef3df9125a30f8fb817fe855b74d7130be45b0ee"},{"sha":"b946f3160eb7953fb30edf1f097ea87be75b33e7","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-20558][CORE] clear InheritableThreadLocal variables in SparkContext when stopping it\n\n## What changes were proposed in this pull request?\n\nTo better understand this problem, let's take a look at an example first:\n```\nobject Main {\n  def main(args: Array[String]): Unit = {\n    var t = new Test\n    new Thread(new Runnable {\n      override def run() = {}\n    }).start()\n    println(\"first thread finished\")\n\n    t.a = null\n    t = new Test\n    new Thread(new Runnable {\n      override def run() = {}\n    }).start()\n  }\n\n}\n\nclass Test {\n  var a = new InheritableThreadLocal[String] {\n    override protected def childValue(parent: String): String = {\n      println(\"parent value is: \" + parent)\n      parent\n    }\n  }\n  a.set(\"hello\")\n}\n```\nThe result is:\n```\nparent value is: hello\nfirst thread finished\nparent value is: hello\nparent value is: hello\n```\n\nOnce an `InheritableThreadLocal` has been set value, child threads will inherit its value as long as it has not been GCed, so setting the variable which holds the `InheritableThreadLocal` to `null` doesn't work as we expected.\n\nIn `SparkContext`, we have an `InheritableThreadLocal` for local properties, we should clear it when stopping `SparkContext`, or all the future child threads will still inherit it and copy the properties and waste memory.\n\nThis is the root cause of https://issues.apache.org/jira/browse/SPARK-20548 , which creates/stops `SparkContext` many times and finally have a lot of `InheritableThreadLocal` alive, and cause OOM when starting new threads in the internal thread pools.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #17833 from cloud-fan/core.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/b946f3160eb7953fb30edf1f097ea87be75b33e7"},{"sha":"6235132a8ce64bb12d825d0a65e5dd052d1ee647","author":{"name":"Michael Armbrust","email":"844aface8de49b32585c3c744b570659f1739c56@databricks.com"},"message":"[SPARK-20567] Lazily bind in GenerateExec\n\nIt is not valid to eagerly bind with the child's output as this causes failures when we attempt to canonicalize the plan (replacing the attribute references with dummies).\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #17838 from marmbrus/fixBindExplode.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/6235132a8ce64bb12d825d0a65e5dd052d1ee647"},{"sha":"db2fb84b4a3c45daa449cc9232340193ce8eb37d","author":{"name":"MechCoder","email":"2c5100b88ec3445ab8c6c52035197fcfe98fba86@gmail.com"},"message":"[SPARK-6227][MLLIB][PYSPARK] Implement PySpark wrappers for SVD and PCA (v2)\n\nAdd PCA and SVD to PySpark's wrappers for `RowMatrix` and `IndexedRowMatrix` (SVD only).\n\nBased on #7963, updated.\n\n## How was this patch tested?\n\nNew doc tests and unit tests. Ran all examples locally.\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\nAuthor: Nick Pentreath <nickp@za.ibm.com>\n\nCloses #17621 from MLnick/SPARK-6227-pyspark-svd-pca.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/db2fb84b4a3c45daa449cc9232340193ce8eb37d"},{"sha":"16fab6b0ef3dcb33f92df30e17680922ad5fb672","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-20523][BUILD] Clean up build warnings for 2.2.0 release\n\n## What changes were proposed in this pull request?\n\nFix build warnings primarily related to Breeze 0.13 operator changes, Java style problems\n\n## How was this patch tested?\n\nExisting tests\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #17803 from srowen/SPARK-20523.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/16fab6b0ef3dcb33f92df30e17680922ad5fb672"},{"sha":"7f96f2d7f2d5abf81dd7f8ca27fea35cf798fd65","author":{"name":"Yan Facai (颜发才)","email":"dea7d1b5b2146d9af7086965f3f8674590ae8d69@gmail.com"},"message":"[SPARK-16957][MLLIB] Use midpoints for split values.\n\n## What changes were proposed in this pull request?\n\nUse midpoints for split values now, and maybe later to make it weighted.\n\n## How was this patch tested?\n\n+ [x] add unit test.\n+ [x] revise Split's unit test.\n\nAuthor: Yan Facai (颜发才) <facai.yan@gmail.com>\nAuthor: 颜发才（Yan Facai） <facai.yan@gmail.com>\n\nCloses #17556 from facaiy/ENH/decision_tree_overflow_and_precision_in_aggregation.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/7f96f2d7f2d5abf81dd7f8ca27fea35cf798fd65"},{"sha":"27f543b15f2f493f6f8373e46b4c9564b0a1bf81","author":{"name":"Liwei Lin","email":"878182135bcb2086b9b1c72277449d6e06bbf76f@gmail.com"},"message":"[SPARK-20441][SPARK-20432][SS] Within the same streaming query, one StreamingRelation should only be transformed to one StreamingExecutionRelation\n\n## What changes were proposed in this pull request?\n\nWithin the same streaming query, when one `StreamingRelation` is referred multiple times – e.g. `df.union(df)` – we should transform it only to one `StreamingExecutionRelation`, instead of two or more different `StreamingExecutionRelation`s (each of which would have a separate set of source, source logs, ...).\n\n## How was this patch tested?\n\nAdded two test cases, each of which would fail without this patch.\n\nAuthor: Liwei Lin <lwlin7@gmail.com>\n\nCloses #17735 from lw-lin/SPARK-20441.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/27f543b15f2f493f6f8373e46b4c9564b0a1bf81"},{"sha":"527fc5d0c990daaacad4740f62cfe6736609b77b","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-20576][SQL] Support generic hint function in Dataset/DataFrame\n\n## What changes were proposed in this pull request?\nWe allow users to specify hints (currently only \"broadcast\" is supported) in SQL and DataFrame. However, while SQL has a standard hint format (/*+ ... */), DataFrame doesn't have one and sometimes users are confused that they can't find how to apply a broadcast hint. This ticket adds a generic hint function on DataFrame that allows using the same hint on DataFrames as well as SQL.\n\nAs an example, after this patch, the following will apply a broadcast hint on a DataFrame using the new hint function:\n\n```\ndf1.join(df2.hint(\"broadcast\"))\n```\n\n## How was this patch tested?\nAdded a test case in DataFrameJoinSuite.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #17839 from rxin/SPARK-20576.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/527fc5d0c990daaacad4740f62cfe6736609b77b"},{"sha":"6b9e49d12fc4c9b29d497122daa4cc9bf4540b16","author":{"name":"Liwei Lin","email":"878182135bcb2086b9b1c72277449d6e06bbf76f@gmail.com"},"message":"[SPARK-19965][SS] DataFrame batch reader may fail to infer partitions when reading FileStreamSink's output\n\n## The Problem\n\nRight now DataFrame batch reader may fail to infer partitions when reading FileStreamSink's output:\n\n```\n[info] - partitioned writing and batch reading with 'basePath' *** FAILED *** (3 seconds, 928 milliseconds)\n[info]   java.lang.AssertionError: assertion failed: Conflicting directory structures detected. Suspicious paths:\n[info] \t***/stream.output-65e3fa45-595a-4d29-b3df-4c001e321637\n[info] \t***/stream.output-65e3fa45-595a-4d29-b3df-4c001e321637/_spark_metadata\n[info]\n[info] If provided paths are partition directories, please set \"basePath\" in the options of the data source to specify the root directory of the table. If there are multiple root directories, please load them separately and then union them.\n[info]   at scala.Predef$.assert(Predef.scala:170)\n[info]   at org.apache.spark.sql.execution.datasources.PartitioningUtils$.parsePartitions(PartitioningUtils.scala:133)\n[info]   at org.apache.spark.sql.execution.datasources.PartitioningUtils$.parsePartitions(PartitioningUtils.scala:98)\n[info]   at org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex.inferPartitioning(PartitioningAwareFileIndex.scala:156)\n[info]   at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.partitionSpec(InMemoryFileIndex.scala:54)\n[info]   at org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex.partitionSchema(PartitioningAwareFileIndex.scala:55)\n[info]   at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:133)\n[info]   at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:361)\n[info]   at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:160)\n[info]   at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:536)\n[info]   at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:520)\n[info]   at org.apache.spark.sql.streaming.FileStreamSinkSuite$$anonfun$8.apply$mcV$sp(FileStreamSinkSuite.scala:292)\n[info]   at org.apache.spark.sql.streaming.FileStreamSinkSuite$$anonfun$8.apply(FileStreamSinkSuite.scala:268)\n[info]   at org.apache.spark.sql.streaming.FileStreamSinkSuite$$anonfun$8.apply(FileStreamSinkSuite.scala:268)\n```\n\n## What changes were proposed in this pull request?\n\nThis patch alters `InMemoryFileIndex` to filter out these `basePath`s whose ancestor is the streaming metadata dir (`_spark_metadata`). E.g., the following and other similar dir or files will be filtered out:\n- (introduced by globbing `basePath/*`)\n   - `basePath/_spark_metadata`\n- (introduced by globbing `basePath/*/*`)\n   - `basePath/_spark_metadata/0`\n   - `basePath/_spark_metadata/1`\n   - ...\n\n## How was this patch tested?\n\nAdded unit tests\n\nAuthor: Liwei Lin <lwlin7@gmail.com>\n\nCloses #17346 from lw-lin/filter-metadata.","distinct":false,"url":"https://api.github.com/repos/zero323/spark/commits/6b9e49d12fc4c9b29d497122daa4cc9bf4540b16"}]}
{"actor":{"display_login":"zero323"}}