{"push_id":1633443439,"size":5,"distinct_size":5,"ref":"refs/heads/master","head":"b0ae6a38a3ef65e4e853781c5127ba38997a8546","before":"82b598b963a21ae9d6a2a9638e86b4165c2a78c9","commits":[{"sha":"12cd00706cbfff4c8ac681fcae65b4c4c8751877","author":{"name":"Sameer Agarwal","email":"a70383dff9f522db5c95b6d04d9bfe45099f5e1d@cs.berkeley.edu"},"message":"[BUILD][MINOR] Fix 2.10 build\n\n## What changes were proposed in this pull request?\n\nhttps://github.com/apache/spark/pull/17385 breaks the 2.10 sbt/maven builds by hitting an empty-string interpolation bug (https://issues.scala-lang.org/browse/SI-7919).\n\nhttps://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/spark-master-compile-sbt-scala-2.10/4072/\nhttps://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/spark-master-compile-maven-scala-2.10/3987/\n\n## How was this patch tested?\n\nCompiles\n\nAuthor: Sameer Agarwal <sameerag@cs.berkeley.edu>\n\nCloses #17391 from sameeragarwal/build-fix.","distinct":true,"url":"https://api.github.com/repos/sureshthalamati/spark/commits/12cd00706cbfff4c8ac681fcae65b4c4c8751877"},{"sha":"07c12c09a75645f6b56b30654455b3838b7b6637","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-18579][SQL] Use ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options in CSV writing\n\n## What changes were proposed in this pull request?\n\nThis PR proposes to support _not_ trimming the white spaces when writing out. These are `false` by default in CSV reading path but these are `true` by default in CSV writing in univocity parser.\n\nBoth `ignoreLeadingWhiteSpace` and `ignoreTrailingWhiteSpace` options are not being used for writing and therefore, we are always trimming the white spaces.\n\nIt seems we should provide a way to keep this white spaces easily.\n\nWIth the data below:\n\n```scala\nval df = spark.read.csv(Seq(\"a , b  , c\").toDS)\ndf.show()\n```\n\n```\n+---+----+---+\n|_c0| _c1|_c2|\n+---+----+---+\n| a | b  |  c|\n+---+----+---+\n```\n\n**Before**\n\n```scala\ndf.write.csv(\"/tmp/text.csv\")\nspark.read.text(\"/tmp/text.csv\").show()\n```\n\n```\n+-----+\n|value|\n+-----+\n|a,b,c|\n+-----+\n```\n\nIt seems this can't be worked around via `quoteAll` too.\n\n```scala\ndf.write.option(\"quoteAll\", true).csv(\"/tmp/text.csv\")\nspark.read.text(\"/tmp/text.csv\").show()\n```\n```\n+-----------+\n|      value|\n+-----------+\n|\"a\",\"b\",\"c\"|\n+-----------+\n```\n\n**After**\n\n```scala\ndf.write.option(\"ignoreLeadingWhiteSpace\", false).option(\"ignoreTrailingWhiteSpace\", false).csv(\"/tmp/text.csv\")\nspark.read.text(\"/tmp/text.csv\").show()\n```\n\n```\n+----------+\n|     value|\n+----------+\n|a , b  , c|\n+----------+\n```\n\nNote that this case is possible in R\n\n```r\n> system(\"cat text.csv\")\nf1,f2,f3\na , b  , c\n> df <- read.csv(file=\"text.csv\")\n> df\n  f1   f2 f3\n1 a   b    c\n> write.csv(df, file=\"text1.csv\", quote=F, row.names=F)\n> system(\"cat text1.csv\")\nf1,f2,f3\na , b  , c\n```\n\n## How was this patch tested?\n\nUnit tests in `CSVSuite` and manual tests for Python.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #17310 from HyukjinKwon/SPARK-18579.","distinct":true,"url":"https://api.github.com/repos/sureshthalamati/spark/commits/07c12c09a75645f6b56b30654455b3838b7b6637"},{"sha":"aefe79890541bc0829f184e03eb3961739ca8ef2","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[MINOR][BUILD] Fix javadoc8 break\n\n## What changes were proposed in this pull request?\n\nSeveral javadoc8 breaks have been introduced. This PR proposes fix those instances so that we can build Scala/Java API docs.\n\n```\n[error] .../spark/sql/core/target/java/org/apache/spark/sql/streaming/GroupState.java:6: error: reference not found\n[error]  * <code>flatMapGroupsWithState</code> operations on {link KeyValueGroupedDataset}.\n[error]                                                             ^\n[error] .../spark/sql/core/target/java/org/apache/spark/sql/streaming/GroupState.java:10: error: reference not found\n[error]  * Both, <code>mapGroupsWithState</code> and <code>flatMapGroupsWithState</code> in {link KeyValueGroupedDataset}\n[error]                                                                                            ^\n[error] .../spark/sql/core/target/java/org/apache/spark/sql/streaming/GroupState.java:51: error: reference not found\n[error]  *    {link GroupStateTimeout.ProcessingTimeTimeout}) or event time (i.e.\n[error]              ^\n[error] .../spark/sql/core/target/java/org/apache/spark/sql/streaming/GroupState.java:52: error: reference not found\n[error]  *    {link GroupStateTimeout.EventTimeTimeout}).\n[error]              ^\n[error] .../spark/sql/core/target/java/org/apache/spark/sql/streaming/GroupState.java:158: error: reference not found\n[error]  *           Spark SQL types (see {link Encoder} for more details).\n[error]                                          ^\n[error] .../spark/mllib/target/java/org/apache/spark/ml/fpm/FPGrowthParams.java:26: error: bad use of '>'\n[error]    * Number of partitions (>=1) used by parallel FP-growth. By default the param is not set, and\n[error]                            ^\n[error] .../spark/sql/core/src/main/java/org/apache/spark/api/java/function/FlatMapGroupsWithStateFunction.java:30: error: reference not found\n[error]  * {link org.apache.spark.sql.KeyValueGroupedDataset#flatMapGroupsWithState(\n[error]           ^\n[error] .../spark/sql/core/target/java/org/apache/spark/sql/KeyValueGroupedDataset.java:211: error: reference not found\n[error]    * See {link GroupState} for more details.\n[error]                 ^\n[error] .../spark/sql/core/target/java/org/apache/spark/sql/KeyValueGroupedDataset.java:232: error: reference not found\n[error]    * See {link GroupState} for more details.\n[error]                 ^\n[error] .../spark/sql/core/target/java/org/apache/spark/sql/KeyValueGroupedDataset.java:254: error: reference not found\n[error]    * See {link GroupState} for more details.\n[error]                 ^\n[error] .../spark/sql/core/target/java/org/apache/spark/sql/KeyValueGroupedDataset.java:277: error: reference not found\n[error]    * See {link GroupState} for more details.\n[error]                 ^\n[error] .../spark/core/target/java/org/apache/spark/TaskContextImpl.java:10: error: reference not found\n[error]  * {link TaskMetrics} &amp; {link MetricsSystem} objects are not thread safe.\n[error]           ^\n[error] .../spark/core/target/java/org/apache/spark/TaskContextImpl.java:10: error: reference not found\n[error]  * {link TaskMetrics} &amp; {link MetricsSystem} objects are not thread safe.\n[error]                                     ^\n[info] 13 errors\n```\n\n```\njekyll 3.3.1 | Error:  Unidoc generation failed\n```\n\n## How was this patch tested?\n\nManually via `jekyll build`\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #17389 from HyukjinKwon/minor-javadoc8-fix.","distinct":true,"url":"https://api.github.com/repos/sureshthalamati/spark/commits/aefe79890541bc0829f184e03eb3961739ca8ef2"},{"sha":"b70c03a42002e924e979acbc98a8b464830be532","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[INFRA] Close stale PRs\n\nCloses #16819\nCloses #13467\nCloses #16083\nCloses #17135\nCloses #8785\nCloses #16278\nCloses #16997\nCloses #17073\nCloses #17220\n\nAdded:\nCloses #12059\nCloses #12524\nCloses #12888\nCloses #16061\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #17386 from srowen/StalePRs.","distinct":true,"url":"https://api.github.com/repos/sureshthalamati/spark/commits/b70c03a42002e924e979acbc98a8b464830be532"},{"sha":"b0ae6a38a3ef65e4e853781c5127ba38997a8546","author":{"name":"Ye Yin","email":"8cd0cc27ad2768b030345f9496f6ec9e9ba86532@qq.com"},"message":"Typo fixup in comment\n\n## What changes were proposed in this pull request?\n\nFixup typo in comment.\n\n## How was this patch tested?\n\nDon't need.\n\nAuthor: Ye Yin <eyniy@qq.com>\n\nCloses #17396 from hustcat/fix.","distinct":true,"url":"https://api.github.com/repos/sureshthalamati/spark/commits/b0ae6a38a3ef65e4e853781c5127ba38997a8546"}]}
{"actor":{"display_login":"sureshthalamati"}}