{"push_id":1610385428,"size":633,"distinct_size":7,"ref":"refs/heads/tmp2","head":"626ee74f5fd623e69a53b663908092afdf478120","before":"d3227dc36ae2807311c7f544c7b386e168cfeab2","commits":[{"sha":"6d2379b3b762cdeff98db5ef4d963135c432580a","author":{"name":"uncleGen","email":"6269a8ea04865dd92a1986593446a029690f118a@gmail.com"},"message":"[SPARK-18485][CORE] Underlying integer overflow when create ChunkedByteBufferOutputStream in MemoryStore\n\n## What changes were proposed in this pull request?\n\nThere is an underlying integer overflow when create ChunkedByteBufferOutputStream in MemoryStore. This PR provide a check before cast.\n\n## How was this patch tested?\n\nadd new unit test\n\nAuthor: uncleGen <hustyugm@gmail.com>\n\nCloses #15915 from uncleGen/SPARK-18485.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/6d2379b3b762cdeff98db5ef4d963135c432580a"},{"sha":"38fd163d0d2c44128bf8872d297b79edd7bd4137","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-18849][ML][SPARKR][DOC] vignettes final check reorg\n\n## What changes were proposed in this pull request?\n\nReorganizing content (copy/paste)\n\n## How was this patch tested?\n\nhttps://felixcheung.github.io/sparkr-vignettes.html\n\nPrevious:\nhttps://felixcheung.github.io/sparkr-vignettes_old.html\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #16301 from felixcheung/rvignettespass2.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/38fd163d0d2c44128bf8872d297b79edd7bd4137"},{"sha":"c0c9e1d27a4c9ede768cfb150cdb26d68472f1da","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-18918][DOC] Missing </td> in Configuration page\n\n### What changes were proposed in this pull request?\nThe configuration page looks messy now, as shown in the nightly build:\nhttps://people.apache.org/~pwendell/spark-nightly/spark-master-docs/latest/configuration.html\n\nStarting from the following location:\n\n![screenshot 2016-12-18 00 26 33](https://cloud.githubusercontent.com/assets/11567269/21292396/ace4719c-c4b8-11e6-8dfd-d9ab95be43d5.png)\n\n### How was this patch tested?\nAttached is the screenshot generated in my local computer after the fix.\n[Configuration - Spark 2.2.0 Documentation.pdf](https://github.com/apache/spark/files/659315/Configuration.-.Spark.2.2.0.Documentation.pdf)\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #16327 from gatorsmile/docFix.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/c0c9e1d27a4c9ede768cfb150cdb26d68472f1da"},{"sha":"1e5c51f336b90cd1eed43e9c6cf00faee696174c","author":{"name":"Yuming Wang","email":"b4045a50547081d83941485c6596bd2032665522@gmail.com"},"message":"[SPARK-18827][CORE] Fix cannot read broadcast on disk\n\n## What changes were proposed in this pull request?\n`NoSuchElementException` will throw since https://github.com/apache/spark/pull/15056 if a broadcast cannot cache in memory. The reason is that that change cannot cover `!unrolled.hasNext` in `next()` function.\n\nThis change is to cover the `!unrolled.hasNext` and check `hasNext` before calling `next` in `blockManager.getLocalValues` to make it  more robust.\n\nWe can cache and read broadcast even it cannot fit in memory from this pull request.\n\nException log:\n```\n16/12/10 10:10:04 INFO UnifiedMemoryManager: Will not store broadcast_131 as the required space (1048576 bytes) exceeds our memory limit (122764 bytes)\n16/12/10 10:10:04 WARN MemoryStore: Failed to reserve initial memory threshold of 1024.0 KB for computing block broadcast_131 in memory.\n16/12/10 10:10:04 WARN MemoryStore: Not enough space to cache broadcast_131 in memory! (computed 384.0 B so far)\n16/12/10 10:10:04 INFO MemoryStore: Memory use = 95.6 KB (blocks) + 0.0 B (scratch space shared across 0 tasks(s)) = 95.6 KB. Storage limit = 119.9 KB.\n16/12/10 10:10:04 ERROR Utils: Exception encountered\njava.util.NoSuchElementException\n\tat org.apache.spark.util.collection.PrimitiveVector$$anon$1.next(PrimitiveVector.scala:58)\n\tat org.apache.spark.storage.memory.PartiallyUnrolledIterator.next(MemoryStore.scala:700)\n\tat org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30)\n\tat org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:210)\n\tat org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:210)\n\tat scala.Option.map(Option.scala:146)\n\tat org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:210)\n\tat org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1269)\n\tat org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:206)\n\tat org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:66)\n\tat org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:66)\n\tat org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:96)\n\tat org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:86)\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:108)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\n16/12/10 10:10:04 ERROR Executor: Exception in task 1.0 in stage 86.0 (TID 134423)\njava.io.IOException: java.util.NoSuchElementException\n\tat org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1276)\n\tat org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:206)\n\tat org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:66)\n\tat org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:66)\n\tat org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:96)\n\tat org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:86)\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:108)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.util.NoSuchElementException\n\tat org.apache.spark.util.collection.PrimitiveVector$$anon$1.next(PrimitiveVector.scala:58)\n\tat org.apache.spark.storage.memory.PartiallyUnrolledIterator.next(MemoryStore.scala:700)\n\tat org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30)\n\tat org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:210)\n\tat org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:210)\n\tat scala.Option.map(Option.scala:146)\n\tat org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:210)\n\tat org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1269)\n\t... 12 more\n```\n\n## How was this patch tested?\n\nAdd unit test\n\nAuthor: Yuming Wang <wgyumg@gmail.com>\n\nCloses #16252 from wangyum/SPARK-18827.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/1e5c51f336b90cd1eed43e9c6cf00faee696174c"},{"sha":"7db09abb0168b77697064c69126ee82ca89609a0","author":{"name":"Zakaria_Hili","email":"ed73de320d07e5a1a1cd4a62dafb038223244e5e@gmail.com"},"message":"[SPARK-18356][ML] KMeans should cache RDD before training\n\n## What changes were proposed in this pull request?\n\nAccording to request of Mr. Joseph Bradley , I did this update of my PR https://github.com/apache/spark/pull/15965 in order to eliminate the extrat fit() method.\n\njkbradley\n## How was this patch tested?\nPass existing tests\n\nAuthor: Zakaria_Hili <zakahili@gmail.com>\nAuthor: HILI Zakaria <zakahili@gmail.com>\n\nCloses #16295 from ZakariaHili/zakbranch.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/7db09abb0168b77697064c69126ee82ca89609a0"},{"sha":"24482858e05bea84cacb41c62be0a9aaa33897ee","author":{"name":"xuanyuanking","email":"64e33dcf1d622ec8e079b74108dbaad04f19fbb4@gmail.com"},"message":"[SPARK-18700][SQL] Add StripedLock for each table's relation in cache\n\n## What changes were proposed in this pull request?\n\nAs the scenario describe in [SPARK-18700](https://issues.apache.org/jira/browse/SPARK-18700), when cachedDataSourceTables invalided, the coming few queries will fetch all FileStatus in listLeafFiles function. In the condition of table has many partitions, these jobs will occupy much memory of driver finally may cause driver OOM.\n\nIn this patch, add StripedLock for each table's relation in cache not for the whole cachedDataSourceTables, each table's load cache operation protected by it.\n\n## How was this patch tested?\n\nAdd a multi-thread access table test in `PartitionedTablePerfStatsSuite` and check it only loading once using metrics in `HiveCatalogMetrics`\n\nAuthor: xuanyuanking <xyliyuanjian@gmail.com>\n\nCloses #16135 from xuanyuanking/SPARK-18700.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/24482858e05bea84cacb41c62be0a9aaa33897ee"},{"sha":"7a75ee1c9224aa5c2e954fe2a71f9ad506f6782b","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-18921][SQL] check database existence with Hive.databaseExists instead of getDatabase\n\n## What changes were proposed in this pull request?\n\nIt's weird that we use `Hive.getDatabase` to check the existence of a database, while Hive has a `databaseExists` interface.\n\nWhat's worse, `Hive.getDatabase` will produce an error message if the database doesn't exist, which is annoying when we only want to check the database existence.\n\nThis PR fixes this and use `Hive.databaseExists` to check database existence.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #16332 from cloud-fan/minor.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/7a75ee1c9224aa5c2e954fe2a71f9ad506f6782b"},{"sha":"70d495dcecce8617b7099fc599fe7c43d7eae66e","author":{"name":"jiangxingbo","email":"5b9d05a325a3f85d4c824395342a210596784248@gmail.com"},"message":"[SPARK-18624][SQL] Implicit cast ArrayType(InternalType)\n\n## What changes were proposed in this pull request?\n\nCurrently `ImplicitTypeCasts` doesn't handle casts between `ArrayType`s, this is not convenient, we should add a rule to enable casting from `ArrayType(InternalType)` to `ArrayType(newInternalType)`.\n\nGoals:\n1. Add a rule to `ImplicitTypeCasts` to enable casting between `ArrayType`s;\n2. Simplify `Percentile` and `ApproximatePercentile`.\n\n## How was this patch tested?\n\nUpdated test cases in `TypeCoercionSuite`.\n\nAuthor: jiangxingbo <jiangxb1987@gmail.com>\n\nCloses #16057 from jiangxb1987/implicit-cast-complex-types.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/70d495dcecce8617b7099fc599fe7c43d7eae66e"},{"sha":"4cb49412d1d7d10ffcc738475928c7de2bc59fd4","author":{"name":"Shivaram Venkataraman","email":"a88946ecf9c024d7b7eca4dac2be9c7e472cd2f5@cs.berkeley.edu"},"message":"[SPARK-18836][CORE] Serialize one copy of task metrics in DAGScheduler\n\n## What changes were proposed in this pull request?\n\nRight now we serialize the empty task metrics once per task – Since this is shared across all tasks we could use the same serialized task metrics across all tasks of a stage.\n\n## How was this patch tested?\n\n- [x] Run tests on EC2 to measure performance improvement\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #16261 from shivaram/task-metrics-one-copy.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/4cb49412d1d7d10ffcc738475928c7de2bc59fd4"},{"sha":"5857b9ac2d9808d9b89a5b29620b5052e2beebf5","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"[SPARK-18928] Check TaskContext.isInterrupted() in FileScanRDD, JDBCRDD & UnsafeSorter\n\n## What changes were proposed in this pull request?\n\nIn order to respond to task cancellation, Spark tasks must periodically check `TaskContext.isInterrupted()`, but this check is missing on a few critical read paths used in Spark SQL, including `FileScanRDD`, `JDBCRDD`, and UnsafeSorter-based sorts. This can cause interrupted / cancelled tasks to continue running and become zombies (as also described in #16189).\n\nThis patch aims to fix this problem by adding `TaskContext.isInterrupted()` checks to these paths. Note that I could have used `InterruptibleIterator` to simply wrap a bunch of iterators but in some cases this would have an adverse performance penalty or might not be effective due to certain special uses of Iterators in Spark SQL. Instead, I inlined `InterruptibleIterator`-style logic into existing iterator subclasses.\n\n## How was this patch tested?\n\nTested manually in `spark-shell` with two different reproductions of non-cancellable tasks, one involving scans of huge files and another involving sort-merge joins that spill to disk. Both causes of zombie tasks are fixed by the changes added here.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #16340 from JoshRosen/sql-task-interruption.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/5857b9ac2d9808d9b89a5b29620b5052e2beebf5"},{"sha":"fa829ce21fb84028d90b739a49c4ece70a17ccfd","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"[SPARK-18761][CORE] Introduce \"task reaper\" to oversee task killing in executors\n\n## What changes were proposed in this pull request?\n\nSpark's current task cancellation / task killing mechanism is \"best effort\" because some tasks may not be interruptible or may not respond to their \"killed\" flags being set. If a significant fraction of a cluster's task slots are occupied by tasks that have been marked as killed but remain running then this can lead to a situation where new jobs and tasks are starved of resources that are being used by these zombie tasks.\n\nThis patch aims to address this problem by adding a \"task reaper\" mechanism to executors. At a high-level, task killing now launches a new thread which attempts to kill the task and then watches the task and periodically checks whether it has been killed. The TaskReaper will periodically re-attempt to call `TaskRunner.kill()` and will log warnings if the task keeps running. I modified TaskRunner to rename its thread at the start of the task, allowing TaskReaper to take a thread dump and filter it in order to log stacktraces from the exact task thread that we are waiting to finish. If the task has not stopped after a configurable timeout then the TaskReaper will throw an exception to trigger executor JVM death, thereby forcibly freeing any resources consumed by the zombie tasks.\n\nThis feature is flagged off by default and is controlled by four new configurations under the `spark.task.reaper.*` namespace. See the updated `configuration.md` doc for details.\n\n## How was this patch tested?\n\nTested via a new test case in `JobCancellationSuite`, plus manual testing.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #16189 from JoshRosen/cancellation.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/fa829ce21fb84028d90b739a49c4ece70a17ccfd"},{"sha":"f923c849e5b8f7e7aeafee59db598a9bf4970f50","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-18899][SPARK-18912][SPARK-18913][SQL] refactor the error checking when append data to an existing table\n\n## What changes were proposed in this pull request?\n\nWhen we append data to an existing table with `DataFrameWriter.saveAsTable`, we will do various checks to make sure the appended data is consistent with the existing data.\n\nHowever, we get the information of the existing table by matching the table relation, instead of looking at the table metadata. This is error-prone, e.g. we only check the number of columns for `HadoopFsRelation`, we forget to check bucketing, etc.\n\nThis PR refactors the error checking by looking at the metadata of the existing table, and fix several bugs:\n* SPARK-18899: We forget to check if the specified bucketing matched the existing table, which may lead to a problematic table that has different bucketing in different data files.\n* SPARK-18912: We forget to check the number of columns for non-file-based data source table\n* SPARK-18913: We don't support append data to a table with special column names.\n\n## How was this patch tested?\nnew regression test.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #16313 from cloud-fan/bug1.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/f923c849e5b8f7e7aeafee59db598a9bf4970f50"},{"sha":"150d26cad435e48441859ff266183161be893fac","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"Tiny style improvement.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/150d26cad435e48441859ff266183161be893fac"},{"sha":"95c95b71ed31b2971475aec6d7776dc234845d0a","author":{"name":"Liang-Chi Hsieh","email":"9f98897f009ec1944a62ee7d389f9f5b1873efea@gmail.com"},"message":"[SPARK-18281] [SQL] [PYSPARK] Remove timeout for reading data through socket for local iterator\n\n## What changes were proposed in this pull request?\n\nThere is a timeout failure when using `rdd.toLocalIterator()` or `df.toLocalIterator()` for a PySpark RDD and DataFrame:\n\n    df = spark.createDataFrame([[1],[2],[3]])\n    it = df.toLocalIterator()\n    row = next(it)\n\n    df2 = df.repartition(1000)  # create many empty partitions which increase materialization time so causing timeout\n    it2 = df2.toLocalIterator()\n    row = next(it2)\n\nThe cause of this issue is, we open a socket to serve the data from JVM side. We set timeout for connection and reading through the socket in Python side. In Python we use a generator to read the data, so we only begin to connect the socket once we start to ask data from it. If we don't consume it immediately, there is connection timeout.\n\nIn the other side, the materialization time for RDD partitions is unpredictable. So we can't set a timeout for reading data through the socket. Otherwise, it is very possibly to fail.\n\n## How was this patch tested?\n\nAdded tests into PySpark.\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #16263 from viirya/fix-pyspark-localiterator.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/95c95b71ed31b2971475aec6d7776dc234845d0a"},{"sha":"caed89321fdabe83e46451ca4e968f86481ad500","author":{"name":"Burak Yavuz","email":"39bd9266538731f16447e7d968c0ba25275ede1e@gmail.com"},"message":"[SPARK-18927][SS] MemorySink for StructuredStreaming can't recover from checkpoint if location is provided in SessionConf\n\n## What changes were proposed in this pull request?\n\nCheckpoint Location can be defined for a StructuredStreaming on a per-query basis by the `DataStreamWriter` options, but it can also be provided through SparkSession configurations. It should be able to recover in both cases when the OutputMode is Complete for MemorySinks.\n\n## How was this patch tested?\n\nUnit tests\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #16342 from brkyvz/chk-rec.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/caed89321fdabe83e46451ca4e968f86481ad500"},{"sha":"047a9d92caa1f0af2305e4afeba8339abf32518b","author":{"name":"Holden Karau","email":"0dd5affe763abbc7f4de3fb3797050edd8c481d3@us.ibm.com"},"message":"[SPARK-18576][PYTHON] Add basic TaskContext information to PySpark\n\n## What changes were proposed in this pull request?\n\nAdds basic TaskContext information to PySpark.\n\n## How was this patch tested?\n\nNew unit tests to `tests.py` & existing unit tests.\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #16211 from holdenk/SPARK-18576-pyspark-taskcontext.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/047a9d92caa1f0af2305e4afeba8339abf32518b"},{"sha":"b2dd8ec6b2c05c996e2d7c0bf8db0073c1ee0b94","author":{"name":"Burak Yavuz","email":"39bd9266538731f16447e7d968c0ba25275ede1e@gmail.com"},"message":"[SPARK-18900][FLAKY-TEST] StateStoreSuite.maintenance\n\n## What changes were proposed in this pull request?\n\nIt was pretty flaky before 10 days ago.\nhttps://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.sql.execution.streaming.state.StateStoreSuite&test_name=maintenance\n\nSince no code changes went into this code path to not be so flaky, I'm just increasing the timeouts such that load related flakiness shouldn't be a problem. As you may see from the testing, I haven't been able to reproduce it.\n\n## How was this patch tested?\n\n2000 retries 5 times\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #16314 from brkyvz/maint-flaky.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/b2dd8ec6b2c05c996e2d7c0bf8db0073c1ee0b94"},{"sha":"24c0c94128770be9034fb69518713d7f6aa1e041","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-18949][SQL] Add recoverPartitions API to Catalog\n\n### What changes were proposed in this pull request?\n\nCurrently, we only have a SQL interface for recovering all the partitions in the directory of a table and update the catalog. `MSCK REPAIR TABLE` or `ALTER TABLE table RECOVER PARTITIONS`. (Actually, very hard for me to remember `MSCK` and have no clue what it means)\n\nAfter the new \"Scalable Partition Handling\", the table repair becomes much more important for making visible the data in the created data source partitioned table.\n\nThus, this PR is to add it into the Catalog interface. After this PR, users can repair the table by\n```Scala\nspark.catalog.recoverPartitions(\"testTable\")\n```\n\n### How was this patch tested?\nModified the existing test cases.\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #16356 from gatorsmile/repairTable.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/24c0c94128770be9034fb69518713d7f6aa1e041"},{"sha":"ba4468bb24f2e13b98e7b66b44203c20393b8ab8","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-18923][DOC][BUILD] Support skipping R/Python API docs\n\n## What changes were proposed in this pull request?\n\nWe can build Python API docs by `cd ./python/docs && make html for Python` and R API docs by `cd ./R && sh create-docs.sh for R` separately. However, `jekyll` fails in some environments.\n\nThis PR aims to support `SKIP_PYTHONDOC` and `SKIP_RDOC` for documentation build in `docs` folder. Currently, we can use `SKIP_SCALADOC` or `SKIP_API`. The reason providing additional options is that the Spark documentation build uses a number of tools to build HTML docs and API docs in Scala, Python and R. Specifically, for Python and R,\n\n- Python API docs requires `sphinx`.\n- R API docs requires `R` installation and `knitr` (and more others libraries).\n\nIn other words, we cannot generate Python API docs without R installation. Also, we cannot generate R API docs without Python `sphinx` installation. If Spark provides `SKIP_PYTHONDOC` and `SKIP_RDOC` like `SKIP_SCALADOC`, it would be more convenient.\n\n## How was this patch tested?\n\nManual.\n\n**Skipping Scala/Java/Python API Doc Build**\n```bash\n$ cd docs\n$ SKIP_SCALADOC=1 SKIP_PYTHONDOC=1 jekyll build\n$ ls api\nDESCRIPTION R\n```\n\n**Skipping Scala/Java/R API Doc Build**\n```bash\n$ cd docs\n$ SKIP_SCALADOC=1 SKIP_RDOC=1 jekyll build\n$ ls api\npython\n```\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #16336 from dongjoon-hyun/SPARK-18923.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/ba4468bb24f2e13b98e7b66b44203c20393b8ab8"},{"sha":"b7650f11c7afbdffc6f5caaafb5dcfd54f7a25ff","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-18947][SQL] SQLContext.tableNames should not call Catalog.listTables\n\n## What changes were proposed in this pull request?\n\nIt's a huge waste to call `Catalog.listTables` in `SQLContext.tableNames`, which only need the table names, while `Catalog.listTables` will get the table metadata for each table name.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #16352 from cloud-fan/minor.","distinct":false,"url":"https://api.github.com/repos/wzhfy/spark/commits/b7650f11c7afbdffc6f5caaafb5dcfd54f7a25ff"}]}
{"actor":{"display_login":"wzhfy"}}