{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16728","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16728/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16728/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16728/events","html_url":"https://github.com/apache/spark/pull/16728","id":203826232,"number":16728,"title":"[SPARK-19359][SQL] Revert Clear useless path after rename a partition with upper-case by HiveExternalCatalog","user":{"login":"gatorsmile","id":11567269,"avatar_url":"https://avatars.githubusercontent.com/u/11567269?v=3","gravatar_id":"","url":"https://api.github.com/users/gatorsmile","html_url":"https://github.com/gatorsmile","followers_url":"https://api.github.com/users/gatorsmile/followers","following_url":"https://api.github.com/users/gatorsmile/following{/other_user}","gists_url":"https://api.github.com/users/gatorsmile/gists{/gist_id}","starred_url":"https://api.github.com/users/gatorsmile/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gatorsmile/subscriptions","organizations_url":"https://api.github.com/users/gatorsmile/orgs","repos_url":"https://api.github.com/users/gatorsmile/repos","events_url":"https://api.github.com/users/gatorsmile/events{/privacy}","received_events_url":"https://api.github.com/users/gatorsmile/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-01-28T19:33:08Z","updated_at":"2017-01-28T21:16:36Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16728","html_url":"https://github.com/apache/spark/pull/16728","diff_url":"https://github.com/apache/spark/pull/16728.diff","patch_url":"https://github.com/apache/spark/pull/16728.patch"},"body":"### What changes were proposed in this pull request?\r\n\r\nThis PR is to revert the changes made in https://github.com/apache/spark/pull/16700. It could cause the data loss after partition rename, because we have a bug in the file renaming. \r\n\r\nNot all the OSs have the same behaviors. For example, on mac OS, if we renaming a path from `.../tbl/a=5/b=6` to `.../tbl/A=5/B=6`. The result is `.../tbl/a=5/B=6`. The expected result is `.../tbl/A=5/B=6`. Thus, renaming on mac OS is not recursive. However, the systems used in Jenkin does not have such an issue. Although this PR is not the root cause, it exposes an existing issue on the code `tablePath.getFileSystem(hadoopConf).rename(wrongPath, rightPath)`\r\n\r\n--- \r\n\r\nHive metastore is not case preserving and keep partition columns with lower case names.\r\n\r\nIf SparkSQL create a table with upper-case partion name use HiveExternalCatalog, when we rename partition, it first call the HiveClient to renamePartition, which will create a new lower case partition path, then SparkSql rename the lower case path to the upper-case.\r\n\r\nwhile if the renamed partition contains more than one depth partition ,e.g. A=1/B=2, hive renamePartition change to a=1/b=2, then SparkSql rename it to A=1/B=2, but the a=1 still exists in the filesystem, we should also delete it.\r\n\r\n### How was this patch tested?\r\nN/A"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/275875221","html_url":"https://github.com/apache/spark/pull/16728#issuecomment-275875221","issue_url":"https://api.github.com/repos/apache/spark/issues/16728","id":275875221,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2017-01-28T21:16:36Z","updated_at":"2017-01-28T21:16:36Z","body":"**[Test build #72110 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72110/testReport)** for PR 16728 at commit [`a398036`](https://github.com/apache/spark/commit/a398036ec957e8d9d67f038095408f576a13c982).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}