{"push_id":1507598290,"size":117,"distinct_size":117,"ref":"refs/heads/master","head":"d06172b88e61c0f79e3dea5703a17c6ae590f248","before":"871f6114ac0075a1b45eda8701113fa20d647de9","commits":[{"sha":"35e974076dcbc5afde8d4259ce88cb5f29d94920","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-19028][SQL] Fixed non-thread-safe functions used in SessionCatalog\n\n### What changes were proposed in this pull request?\nFixed non-thread-safe functions used in SessionCatalog:\n- refreshTable\n- lookupRelation\n\n### How was this patch tested?\nN/A\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #16437 from gatorsmile/addSyncToLookUpTable.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/35e974076dcbc5afde8d4259ce88cb5f29d94920"},{"sha":"2394047370d2d93bd8bc57b996fee47465c470af","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-19050][SS][TESTS] Fix EventTimeWatermarkSuite 'delay in months and years handled correctly'\n\n## What changes were proposed in this pull request?\n\n`monthsSinceEpoch` in this test is like `math.floor(num)`, so `monthDiff` has two possible values.\n\n## How was this patch tested?\n\nJenkins.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #16449 from zsxwing/watermark-test-hotfix.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/2394047370d2d93bd8bc57b996fee47465c470af"},{"sha":"ba4881268e0c3b9dcf99b3077e6394259c705d55","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[BUILD] Close stale PRs\n\nCloses #12968\nCloses #16215\nCloses #16212\nCloses #16086\nCloses #15713\nCloses #16413\nCloses #16396\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #16447 from srowen/CloseStalePRs.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/ba4881268e0c3b9dcf99b3077e6394259c705d55"},{"sha":"808b84e2de3537a47dc1c5f426a89dd4ec6bf0c5","author":{"name":"Liwei Lin","email":"878182135bcb2086b9b1c72277449d6e06bbf76f@gmail.com"},"message":"[SPARK-19041][SS] Fix code snippet compilation issues in Structured Streaming Programming Guide\n\n## What changes were proposed in this pull request?\n\nCurrently some code snippets in the programming guide just do not compile. We should fix them.\n\n## How was this patch tested?\n\n```\nSKIP_API=1 jekyll build\n```\n\n## Screenshot from part of the change:\n\n![snip20161231_37](https://cloud.githubusercontent.com/assets/15843379/21576864/cc52fcd8-cf7b-11e6-8bd6-f935d9ff4a6b.png)\n\nAuthor: Liwei Lin <lwlin7@gmail.com>\n\nCloses #16442 from lw-lin/ss-pro-guide-.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/808b84e2de3537a47dc1c5f426a89dd4ec6bf0c5"},{"sha":"0ac2f1e71f62ec925ed0e19c4654759d155efc35","author":{"name":"Liang-Chi Hsieh","email":"9f98897f009ec1944a62ee7d389f9f5b1873efea@gmail.com"},"message":"[MINOR][DOC] Minor doc change for YARN credential providers\n\n## What changes were proposed in this pull request?\n\nThe configuration `spark.yarn.security.tokens.{service}.enabled` is deprecated. Now we should use `spark.yarn.security.credentials.{service}.enabled`. Some places in the doc is not updated yet.\n\n## How was this patch tested?\n\nN/A. Just doc change.\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #16444 from viirya/minor-credential-provider-doc.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/0ac2f1e71f62ec925ed0e19c4654759d155efc35"},{"sha":"f1330b1d9e7b1d5de611e59eecae1bf0b0616d81","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-19022][TESTS] Fix tests dependent on OS due to different newline characters\n\n## What changes were proposed in this pull request?\n\nThere are two tests failing on Windows due to the different newlines.\n\n```\n - StreamingQueryProgress - prettyJson *** FAILED *** (0 milliseconds)\n \"{\n    \"id\" : \"39788670-6722-48b7-a248-df6ba08722ac\",\n    \"runId\" : \"422282f1-3b81-4b47-a15d-82dda7e69390\",\n    \"name\" : \"myName\",\n    ...\n  }\" did not equal \"{\n    \"id\" : \"39788670-6722-48b7-a248-df6ba08722ac\",\n    \"runId\" : \"422282f1-3b81-4b47-a15d-82dda7e69390\",\n    \"name\" : \"myName\",\n    ...\n  }\"\n  ...\n```\n\n```\n - StreamingQueryStatus - prettyJson *** FAILED *** (0 milliseconds)\n \"{\n    \"message\" : \"active\",\n    \"isDataAvailable\" : true,\n    \"isTriggerActive\" : false\n  }\" did not equal \"{\n    \"message\" : \"active\",\n    \"isDataAvailable\" : true,\n    \"isTriggerActive\" : false\n  }\"\n  ...\n```\n\nThe reason is, `pretty` in `org.json4s.pretty` writes OS-dependent newlines but the string defined in the tests are `\\n`. This ends up with test failures.\n\nThis PR proposes to compare these regardless of newline concerns.\n\n## How was this patch tested?\n\nManually tested via AppVeyor.\n\n**Before**\nhttps://ci.appveyor.com/project/spark-test/spark/build/417-newlines-fix-before\n\n**After**\nhttps://ci.appveyor.com/project/spark-test/spark/build/418-newlines-fix\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #16433 from HyukjinKwon/tests-StreamingQueryStatusAndProgressSuite.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/f1330b1d9e7b1d5de611e59eecae1bf0b0616d81"},{"sha":"46b212602428f1f11c184c836b4e09c150d0ee30","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-19002][BUILD][PYTHON] Check pep8 against all Python scripts\n\n## What changes were proposed in this pull request?\n\nThis PR proposes to check pep8 against all other Python scripts and fix the errors as below:\n\n```bash\n./dev/create-release/generate-contributors.py\n./dev/create-release/releaseutils.py\n./dev/create-release/translate-contributors.py\n./dev/lint-python\n./python/docs/epytext.py\n./examples/src/main/python/mllib/decision_tree_classification_example.py\n./examples/src/main/python/mllib/decision_tree_regression_example.py\n./examples/src/main/python/mllib/gradient_boosting_classification_example.py\n./examples/src/main/python/mllib/gradient_boosting_regression_example.py\n./examples/src/main/python/mllib/linear_regression_with_sgd_example.py\n./examples/src/main/python/mllib/logistic_regression_with_lbfgs_example.py\n./examples/src/main/python/mllib/naive_bayes_example.py\n./examples/src/main/python/mllib/random_forest_classification_example.py\n./examples/src/main/python/mllib/random_forest_regression_example.py\n./examples/src/main/python/mllib/svm_with_sgd_example.py\n./examples/src/main/python/streaming/network_wordjoinsentiments.py\n./sql/hive/src/test/resources/data/scripts/cat.py\n./sql/hive/src/test/resources/data/scripts/cat_error.py\n./sql/hive/src/test/resources/data/scripts/doubleescapedtab.py\n./sql/hive/src/test/resources/data/scripts/dumpdata_script.py\n./sql/hive/src/test/resources/data/scripts/escapedcarriagereturn.py\n./sql/hive/src/test/resources/data/scripts/escapednewline.py\n./sql/hive/src/test/resources/data/scripts/escapedtab.py\n./sql/hive/src/test/resources/data/scripts/input20_script.py\n./sql/hive/src/test/resources/data/scripts/newline.py\n```\n\n## How was this patch tested?\n\n- `./python/docs/epytext.py`\n\n  ```bash\n  cd ./python/docs $$ make html\n  ```\n\n- pep8 check (Python 2.7 / Python 3.3.6)\n\n  ```\n  ./dev/lint-python\n  ```\n\n- `./dev/merge_spark_pr.py` (Python 2.7 only / Python 3.3.6 not working)\n\n  ```bash\n  python -m doctest -v ./dev/merge_spark_pr.py\n  ```\n\n- `./dev/create-release/releaseutils.py` `./dev/create-release/generate-contributors.py` `./dev/create-release/translate-contributors.py` (Python 2.7 only / Python 3.3.6 not working)\n\n  ```bash\n  python generate-contributors.py\n  python translate-contributors.py\n  ```\n\n- Examples (Python 2.7 / Python 3.3.6)\n\n  ```bash\n  ./bin/spark-submit examples/src/main/python/mllib/decision_tree_classification_example.py\n  ./bin/spark-submit examples/src/main/python/mllib/decision_tree_regression_example.py\n  ./bin/spark-submit examples/src/main/python/mllib/gradient_boosting_classification_example.py\n  ./bin/spark-submit examples/src/main/python/mllib/gradient_boosting_regression_example.p\n  ./bin/spark-submit examples/src/main/python/mllib/random_forest_classification_example.py\n  ./bin/spark-submit examples/src/main/python/mllib/random_forest_regression_example.py\n  ```\n\n- Examples (Python 2.7 only / Python 3.3.6 not working)\n  ```\n  ./bin/spark-submit examples/src/main/python/mllib/linear_regression_with_sgd_example.py\n  ./bin/spark-submit examples/src/main/python/mllib/logistic_regression_with_lbfgs_example.py\n  ./bin/spark-submit examples/src/main/python/mllib/naive_bayes_example.py\n  ./bin/spark-submit examples/src/main/python/mllib/svm_with_sgd_example.py\n  ```\n\n- `sql/hive/src/test/resources/data/scripts/*.py` (Python 2.7 / Python 3.3.6 within suggested changes)\n\n  Manually tested only changed ones.\n\n- `./dev/github_jira_sync.py` (Python 2.7 only / Python 3.3.6 not working)\n\n  Manually tested this after disabling actually adding comments and links.\n\nAnd also via Jenkins tests.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #16405 from HyukjinKwon/minor-pep8.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/46b212602428f1f11c184c836b4e09c150d0ee30"},{"sha":"a6cd9dbc6095570e93dab1d93671abecdce40c25","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-19029][SQL] Remove databaseName from SimpleCatalogRelation\n\n### What changes were proposed in this pull request?\nRemove useless `databaseName ` from `SimpleCatalogRelation`.\n\n### How was this patch tested?\nExisting test cases.\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #16438 from gatorsmile/removeDBFromSimpleCatalogRelation.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/a6cd9dbc6095570e93dab1d93671abecdce40c25"},{"sha":"ae83c211257c508989c703d54f2aeec8b2b5f14d","author":{"name":"Zhenhua Wang","email":"e1bb7e32d2908d1daa3f64ed12aed842612a516d@163.com"},"message":"[SPARK-18998][SQL] Add a cbo conf to switch between default statistics and estimated statistics\n\n## What changes were proposed in this pull request?\n\nWe add a cbo configuration to switch between default stats and estimated stats.\nWe also define a new statistics method `planStats` in LogicalPlan with conf as its parameter, in order to pass the cbo switch and other estimation related configurations in the future. `planStats` is used on the caller sides (i.e. in Optimizer and Strategies) to make transformation decisions based on stats.\n\n## How was this patch tested?\n\nAdd a test case using a dummy LogicalPlan.\n\nAuthor: Zhenhua Wang <wzh_zju@163.com>\n\nCloses #16401 from wzhfy/cboSwitch.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/ae83c211257c508989c703d54f2aeec8b2b5f14d"},{"sha":"e5c307c50a660f706799f1f7f6890bcec888d96b","author":{"name":"Weiqing Yang","email":"c0c88f1153e01312d8f77a3074d15074c5d139c9@gmail.com"},"message":"[MINOR] Add missing sc.stop() to end of examples\n\n## What changes were proposed in this pull request?\n\nAdd `finally` clause for `sc.stop()` in the `test(\"register and deregister Spark listener from SparkContext\")`.\n\n## How was this patch tested?\nPass the build and unit tests.\n\nAuthor: Weiqing Yang <yangweiqing001@gmail.com>\n\nCloses #16426 from weiqingy/testIssue.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/e5c307c50a660f706799f1f7f6890bcec888d96b"},{"sha":"52636226dc8cb7fcf00381d65e280d651b25a382","author":{"name":"Liang-Chi Hsieh","email":"9f98897f009ec1944a62ee7d389f9f5b1873efea@gmail.com"},"message":"[SPARK-18932][SQL] Support partial aggregation for collect_set/collect_list\n\n## What changes were proposed in this pull request?\n\nCurrently collect_set/collect_list aggregation expression don't support partial aggregation. This patch is to enable partial aggregation for them.\n\n## How was this patch tested?\n\nJenkins tests.\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #16371 from viirya/collect-partial-support.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/52636226dc8cb7fcf00381d65e280d651b25a382"},{"sha":"7a2b5f93bc3d3224470837ed3323964ba7cb1dca","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-18877][SQL] `CSVInferSchema.inferField` on DecimalType should find a common type with `typeSoFar`\n\n## What changes were proposed in this pull request?\n\nCSV type inferencing causes `IllegalArgumentException` on decimal numbers with heterogeneous precisions and scales because the current logic uses the last decimal type in a **partition**. Specifically, `inferRowType`, the **seqOp** of **aggregate**, returns the last decimal type. This PR fixes it to use `findTightestCommonType`.\n\n**decimal.csv**\n```\n9.03E+12\n1.19E+11\n```\n\n**BEFORE**\n```scala\nscala> spark.read.format(\"csv\").option(\"inferSchema\", true).load(\"decimal.csv\").printSchema\nroot\n |-- _c0: decimal(3,-9) (nullable = true)\n\nscala> spark.read.format(\"csv\").option(\"inferSchema\", true).load(\"decimal.csv\").show\n16/12/16 14:32:49 ERROR Executor: Exception in task 0.0 in stage 4.0 (TID 4)\njava.lang.IllegalArgumentException: requirement failed: Decimal precision 4 exceeds max precision 3\n```\n\n**AFTER**\n```scala\nscala> spark.read.format(\"csv\").option(\"inferSchema\", true).load(\"decimal.csv\").printSchema\nroot\n |-- _c0: decimal(4,-9) (nullable = true)\n\nscala> spark.read.format(\"csv\").option(\"inferSchema\", true).load(\"decimal.csv\").show\n+---------+\n|      _c0|\n+---------+\n|9.030E+12|\n| 1.19E+11|\n+---------+\n```\n\n## How was this patch tested?\n\nPass the newly add test case.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #16320 from dongjoon-hyun/SPARK-18877.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/7a2b5f93bc3d3224470837ed3323964ba7cb1dca"},{"sha":"89bf370e4f53c02b018b23adc653cd718869489e","author":{"name":"Devaraj K","email":"b8ebf05c6cb8ebae39c552537bfd7d84ebc26f96@apache.org"},"message":"[SPARK-15555][MESOS] Driver with --supervise option cannot be killed in Mesos mode\n\n## What changes were proposed in this pull request?\n\nNot adding the Killed applications for retry.\n## How was this patch tested?\n\nI have verified manually in the Mesos cluster, with the changes the killed applications move to Finished Drivers section and will not retry.\n\nAuthor: Devaraj K <devaraj@apache.org>\n\nCloses #13323 from devaraj-kavali/SPARK-15555.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/89bf370e4f53c02b018b23adc653cd718869489e"},{"sha":"b67b35f76b684c5176dc683e7491fd01b43f4467","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-19048][SQL] Delete Partition Location when Dropping Managed Partitioned Tables in InMemoryCatalog\n\n### What changes were proposed in this pull request?\nThe data in the managed table should be deleted after table is dropped. However, if the partition location is not under the location of the partitioned table, it is not deleted as expected. Users can specify any location for the partition when they adding a partition.\n\nThis PR is to delete partition location when dropping managed partitioned tables stored in `InMemoryCatalog`.\n\n### How was this patch tested?\nAdded test cases for both HiveExternalCatalog and InMemoryCatalog\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #16448 from gatorsmile/unsetSerdeProp.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/b67b35f76b684c5176dc683e7491fd01b43f4467"},{"sha":"cbd11d235752d0ab30cfdbf2351cb3e68a123606","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-19072][SQL] codegen of Literal should not output boxed value\n\n## What changes were proposed in this pull request?\n\nIn https://github.com/apache/spark/pull/16402 we made a mistake that, when double/float is infinity, the `Literal` codegen will output boxed value and cause wrong result.\n\nThis PR fixes this by special handling infinity to not output boxed value.\n\n## How was this patch tested?\n\nnew regression test\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #16469 from cloud-fan/literal.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/cbd11d235752d0ab30cfdbf2351cb3e68a123606"},{"sha":"fe1c895e16c475a6f271ce600a42a8d0dc7986e5","author":{"name":"mingfei","email":"6f3bf979535db71cc7e93f00a266507a61811ed4@alipay.com"},"message":"[SPARK-19073] LauncherState should be only set to SUBMITTED after the application is submitted\n\n## What changes were proposed in this pull request?\nLauncherState should be only set to SUBMITTED after the application is submitted.\nCurrently the state is set before the application is actually submitted.\n\n## How was this patch tested?\nno test is added in this patch\n\nAuthor: mingfei <mingfei.smf@alipay.com>\n\nCloses #16459 from shimingfei/fixLauncher.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/fe1c895e16c475a6f271ce600a42a8d0dc7986e5"},{"sha":"101556d0fa704deca0f4a2e5070906d4af2c861b","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-19060][SQL] remove the supportsPartial flag in AggregateFunction\n\n## What changes were proposed in this pull request?\n\nNow all aggregation functions support partial aggregate, we can remove the `supportsPartual` flag in `AggregateFunction`\n\n## How was this patch tested?\n\nexisting tests.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #16461 from cloud-fan/partial.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/101556d0fa704deca0f4a2e5070906d4af2c861b"},{"sha":"7a82505817d479007adff6424473063d2003fcc1","author":{"name":"Zheng RuiFeng","email":"688fbc7eb0ddb0dce0ccb704b29547c7b3cab719@foxmail.com"},"message":"[SPARK-19054][ML] Eliminate extra pass in NB\n\n## What changes were proposed in this pull request?\neliminate unnecessary extra pass in NB's train\n\n## How was this patch tested?\nexisting tests\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #16453 from zhengruifeng/nb_getNC.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/7a82505817d479007adff6424473063d2003fcc1"},{"sha":"a1e40b1f5d651305bbd0ba05779263a44f607498","author":{"name":"Niranjan Padmanabhan","email":"9e09bcd0cbcdcfcd984cc5a9d5c00e3dad1277eb@gmail.com"},"message":"[MINOR][DOCS] Remove consecutive duplicated words/typo in Spark Repo\n\n## What changes were proposed in this pull request?\nThere are many locations in the Spark repo where the same word occurs consecutively. Sometimes they are appropriately placed, but many times they are not. This PR removes the inappropriately duplicated words.\n\n## How was this patch tested?\nN/A since only docs or comments were updated.\n\nAuthor: Niranjan Padmanabhan <niranjan.padmanabhan@gmail.com>\n\nCloses #16455 from neurons/np.structure_streaming_doc.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/a1e40b1f5d651305bbd0ba05779263a44f607498"},{"sha":"4262fb0d55aed1a023e1813e09deefda8a7ce26b","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@databricks.com"},"message":"[SPARK-19070] Clean-up dataset actions\n\n## What changes were proposed in this pull request?\nDataset actions currently spin off a new `Dataframe` only to track query execution. This PR simplifies this code path by using the `Dataset.queryExecution` directly. This PR also merges the typed and untyped action evaluation paths.\n\n## How was this patch tested?\nExisting tests.\n\nAuthor: Herman van Hovell <hvanhovell@databricks.com>\n\nCloses #16466 from hvanhovell/SPARK-19070.","distinct":true,"url":"https://api.github.com/repos/traviscrawford/spark/commits/4262fb0d55aed1a023e1813e09deefda8a7ce26b"}]}
{"actor":{"display_login":"traviscrawford"}}