{"push_id":1974628257,"size":17,"distinct_size":2,"ref":"refs/heads/SPARK-20682-3","head":"ed43eb7fb47a1c65875bbf97a8e108abfb115925","before":"27d24a8a9c5eec9242cdcdaa37af2e15f3f2e6a7","commits":[{"sha":"b9ab791a9efb0dc165ba283c91acf831fa6be5d8","author":{"name":"Sanket Chintapalli","email":"d22deb47df1d7fc05ea2335d99a9dc864f69f57c@yahoo-inc.com"},"message":"[SPARK-21890] Credentials not being passed to add the tokens\n\nI observed this while running a oozie job trying to connect to hbase via spark.\nIt look like the creds are not being passed in thehttps://github.com/apache/spark/blob/branch-2.2/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala#L53 for 2.2 release.\nMore Info as to why it fails on secure grid:\nOozie client gets the necessary tokens the application needs before launching. It passes those tokens along to the oozie launcher job (MR job) which will then actually call the Spark client to launch the spark app and pass the tokens along.\nThe oozie launcher job cannot get anymore tokens because all it has is tokens ( you can't get tokens with tokens, you need tgt or keytab).\nThe error here is because the launcher job runs the Spark Client to submit the spark job but the spark client doesn't see that it already has the hdfs tokens so it tries to get more, which ends with the exception.\nThere was a change with SPARK-19021 to generalize the hdfs credentials provider that changed it so we don't pass the existing credentials into the call to get tokens so it doesn't realize it already has the necessary tokens.\n\nhttps://issues.apache.org/jira/browse/SPARK-21890\nModified to pass creds to get delegation tokens\n\nAuthor: Sanket Chintapalli <schintap@yahoo-inc.com>\n\nCloses #19140 from redsanket/SPARK-21890-master.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/b9ab791a9efb0dc165ba283c91acf831fa6be5d8"},{"sha":"e00f1a1da12be4a1fdb7b89eb5e098aa16c5c2c3","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadata from SQLConf and docs\n\n## What changes were proposed in this pull request?\n\nSince [SPARK-15639](https://github.com/apache/spark/pull/13701), `spark.sql.parquet.cacheMetadata` and `PARQUET_CACHE_METADATA` is not used. This PR removes from SQLConf and docs.\n\n## How was this patch tested?\n\nPass the existing Jenkins.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #19129 from dongjoon-hyun/SPARK-13656.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/e00f1a1da12be4a1fdb7b89eb5e098aa16c5c2c3"},{"sha":"c26976fe148a2a59cec2f399484be73d08fb6b7f","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-21939][TEST] Use TimeLimits instead of Timeouts\n\nSince ScalaTest 3.0.0, `org.scalatest.concurrent.Timeouts` is deprecated.\nThis PR replaces the deprecated one with `org.scalatest.concurrent.TimeLimits`.\n\n```scala\n-import org.scalatest.concurrent.Timeouts._\n+import org.scalatest.concurrent.TimeLimits._\n```\n\nPass the existing test suites.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #19150 from dongjoon-hyun/SPARK-21939.\n\nChange-Id: I1a1b07f1b97e51e2263dfb34b7eaaa099b2ded5e","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/c26976fe148a2a59cec2f399484be73d08fb6b7f"},{"sha":"57bc1e9eb452284cbed090dbd5008eb2062f1b36","author":{"name":"Takuya UESHIN","email":"9d9445011f9a909d02246cb8499bd5adcacb7b63@databricks.com"},"message":"[SPARK-21950][SQL][PYTHON][TEST] pyspark.sql.tests.SQLTests2 should stop SparkContext.\n\n## What changes were proposed in this pull request?\n\n`pyspark.sql.tests.SQLTests2` doesn't stop newly created spark context in the test and it might affect the following tests.\nThis pr makes `pyspark.sql.tests.SQLTests2` stop `SparkContext`.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Takuya UESHIN <ueshin@databricks.com>\n\nCloses #19158 from ueshin/issues/SPARK-21950.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/57bc1e9eb452284cbed090dbd5008eb2062f1b36"},{"sha":"f62b20f39c5e44ad6de535117e076060fef3f9ec","author":{"name":"liuxian","email":"d698299a02cb9d6095b897cb57e6057a3afd73b5@zte.com.cn"},"message":"[SPARK-21949][TEST] Tables created in unit tests should be dropped after use\n\n## What changes were proposed in this pull request?\n Tables should be dropped after use in unit tests.\n## How was this patch tested?\nN/A\n\nAuthor: liuxian <liu.xian3@zte.com.cn>\n\nCloses #19155 from 10110346/droptable.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/f62b20f39c5e44ad6de535117e076060fef3f9ec"},{"sha":"6e37524a1fd26bbfe5034ecf971472931d1d47a9","author":{"name":"Liang-Chi Hsieh","email":"9f98897f009ec1944a62ee7d389f9f5b1873efea@gmail.com"},"message":"[SPARK-21726][SQL] Check for structural integrity of the plan in Optimzer in test mode.\n\n## What changes were proposed in this pull request?\n\nWe have many optimization rules now in `Optimzer`. Right now we don't have any checks in the optimizer to check for the structural integrity of the plan (e.g. resolved). When debugging, it is difficult to identify which rules return invalid plans.\n\nIt would be great if in test mode, we can check whether a plan is still resolved after the execution of each rule, so we can catch rules that return invalid plans.\n\n## How was this patch tested?\n\nAdded tests.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #18956 from viirya/SPARK-21726.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/6e37524a1fd26bbfe5034ecf971472931d1d47a9"},{"sha":"dbb824125d4d31166d9a47c330f8d51f5d159515","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-21936][SQL] backward compatibility test framework for HiveExternalCatalog\n\n## What changes were proposed in this pull request?\n\n`HiveExternalCatalog` is a semi-public interface. When creating tables, `HiveExternalCatalog` converts the table metadata to hive table format and save into hive metastore. It's very import to guarantee backward compatibility here, i.e., tables created by previous Spark versions should still be readable in newer Spark versions.\n\nPreviously we find backward compatibility issues manually, which is really easy to miss bugs. This PR introduces a test framework to automatically test `HiveExternalCatalog` backward compatibility, by downloading Spark binaries with different versions, and create tables with these Spark versions, and read these tables with current Spark version.\n\n## How was this patch tested?\n\ntest-only change\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #19148 from cloud-fan/test.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/dbb824125d4d31166d9a47c330f8d51f5d159515"},{"sha":"0dfc1ec59e45c836cb968bc9b77c69bf0e917b06","author":{"name":"Liang-Chi Hsieh","email":"9f98897f009ec1944a62ee7d389f9f5b1873efea@gmail.com"},"message":"[SPARK-21726][SQL][FOLLOW-UP] Check for structural integrity of the plan in Optimzer in test mode\n\n## What changes were proposed in this pull request?\n\nThe condition in `Optimizer.isPlanIntegral` is wrong. We should always return `true` if not in test mode.\n\n## How was this patch tested?\n\nManually test.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #19161 from viirya/SPARK-21726-followup.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/0dfc1ec59e45c836cb968bc9b77c69bf0e917b06"},{"sha":"8a4f228dc0afed7992695486ecab6bc522f1e392","author":{"name":"Kazuaki Ishizaki","email":"64981c6bd46971e76f308b8a384367244e8169bf@jp.ibm.com"},"message":"[SPARK-21946][TEST] fix flaky test: \"alter table: rename cached table\" in InMemoryCatalogedDDLSuite\n\n## What changes were proposed in this pull request?\n\nThis PR fixes flaky test `InMemoryCatalogedDDLSuite \"alter table: rename cached table\"`.\nSince this test validates distributed DataFrame, the result should be checked by using `checkAnswer`. The original version used `df.collect().Seq` method that does not guaranty an order of each element of the result.\n\n## How was this patch tested?\n\nUse existing test case\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #19159 from kiszk/SPARK-21946.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/8a4f228dc0afed7992695486ecab6bc522f1e392"},{"sha":"8598d03a00a39dd23646bf752f9fed5d28e271c6","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-15243][ML][SQL][PYTHON] Add missing support for unicode in Param methods & functions in dataframe\n\n## What changes were proposed in this pull request?\n\nThis PR proposes to support unicodes in Param methods in ML, other missed functions in DataFrame.\n\nFor example, this causes a `ValueError` in Python 2.x when param is a unicode string:\n\n```python\n>>> from pyspark.ml.classification import LogisticRegression\n>>> lr = LogisticRegression()\n>>> lr.hasParam(\"threshold\")\nTrue\n>>> lr.hasParam(u\"threshold\")\nTraceback (most recent call last):\n ...\n    raise TypeError(\"hasParam(): paramName must be a string\")\nTypeError: hasParam(): paramName must be a string\n```\n\nThis PR is based on https://github.com/apache/spark/pull/13036\n\n## How was this patch tested?\n\nUnit tests in `python/pyspark/ml/tests.py` and `python/pyspark/sql/tests.py`.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\nAuthor: sethah <seth.hendrickson16@gmail.com>\n\nCloses #17096 from HyukjinKwon/SPARK-15243.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/8598d03a00a39dd23646bf752f9fed5d28e271c6"},{"sha":"31c74fec24ae3bc8b9eb4ecd90896de459c3cc22","author":{"name":"Xin Ren","email":"82e6f9780faa35cac922f524c8a765f69b3ff7ff@126.com"},"message":"[SPARK-19866][ML][PYSPARK] Add local version of Word2Vec findSynonyms for spark.ml: Python API\n\nhttps://issues.apache.org/jira/browse/SPARK-19866\n\n## What changes were proposed in this pull request?\n\nAdd Python API for findSynonymsArray matching Scala API.\n\n## How was this patch tested?\n\nManual test\n`./python/run-tests --python-executables=python2.7 --modules=pyspark-ml`\n\nAuthor: Xin Ren <iamshrek@126.com>\nAuthor: Xin Ren <renxin.ubc@gmail.com>\nAuthor: Xin Ren <keypointt@users.noreply.github.com>\n\nCloses #17451 from keypointt/SPARK-19866.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/31c74fec24ae3bc8b9eb4ecd90896de459c3cc22"},{"sha":"8a5eb5068104f527426fb2d0908f45c8eff0749f","author":{"name":"Andrew Ash","email":"6b7c097d961f68685fc978cf91eeae17be6cbb45@andrewash.com"},"message":"[SPARK-21941] Stop storing unused attemptId in SQLTaskMetrics\n\n## What changes were proposed in this pull request?\n\nIn a driver heap dump containing 390,105 instances of SQLTaskMetrics this\nwould have saved me approximately 3.2MB of memory.\n\nSince we're not getting any benefit from storing this unused value, let's\neliminate it until a future PR makes use of it.\n\n## How was this patch tested?\n\nExisting unit tests\n\nAuthor: Andrew Ash <andrew@andrewash.com>\n\nCloses #19153 from ash211/aash/trim-sql-listener.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/8a5eb5068104f527426fb2d0908f45c8eff0749f"},{"sha":"6b45d7e941eba8a36be26116787322d9e3ae25d0","author":{"name":"Liang-Chi Hsieh","email":"9f98897f009ec1944a62ee7d389f9f5b1873efea@gmail.com"},"message":"[SPARK-21954][SQL] JacksonUtils should verify MapType's value type instead of key type\n\n## What changes were proposed in this pull request?\n\n`JacksonUtils.verifySchema` verifies if a data type can be converted to JSON. For `MapType`, it now verifies the key type. However, in `JacksonGenerator`, when converting a map to JSON, we only care about its values and create a writer for the values. The keys in a map are treated as strings by calling `toString` on the keys.\n\nThus, we should change `JacksonUtils.verifySchema` to verify the value type of `MapType`.\n\n## How was this patch tested?\n\nAdded tests.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #19167 from viirya/test-jacksonutils.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/6b45d7e941eba8a36be26116787322d9e3ae25d0"},{"sha":"e4d8f9a36ac27b0175f310bf5592b2881b025468","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[MINOR][SQL] Correct DataFrame doc.\n\n## What changes were proposed in this pull request?\nCorrect DataFrame doc.\n\n## How was this patch tested?\nOnly doc change, no tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #19173 from yanboliang/df-doc.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/e4d8f9a36ac27b0175f310bf5592b2881b025468"},{"sha":"f76790557b063edc3080d5c792167e2f8b7060d1","author":{"name":"Jane Wang","email":"fded6a73aa68ed0c132d88900db64f549308e49d@fb.com"},"message":"[SPARK-4131] Support \"Writing data into the filesystem from queries\"\n\n## What changes were proposed in this pull request?\n\nThis PR implements the sql feature:\nINSERT OVERWRITE [LOCAL] DIRECTORY directory1\n  [ROW FORMAT row_format] [STORED AS file_format]\n  SELECT ... FROM ...\n\n## How was this patch tested?\nAdded new unittests and also pulled the code to fb-spark so that we could test writing to hdfs directory.\n\nAuthor: Jane Wang <janewang@fb.com>\n\nCloses #18975 from janewangfb/port_local_directory.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/f76790557b063edc3080d5c792167e2f8b7060d1"},{"sha":"9317a375f261980cf5753fd6be0ce8377d71958d","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-20682][SQL] Update ORC data source based on Apache ORC library","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/9317a375f261980cf5753fd6be0ce8377d71958d"},{"sha":"ed43eb7fb47a1c65875bbf97a8e108abfb115925","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"merge SPARK-21791 into here.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/ed43eb7fb47a1c65875bbf97a8e108abfb115925"}]}
{"actor":{"display_login":"dongjoon-hyun"}}