{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/10498","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/10498/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/10498/comments","events_url":"https://api.github.com/repos/apache/spark/issues/10498/events","html_url":"https://github.com/apache/spark/pull/10498","id":124074637,"number":10498,"title":"[SPARK-12539][SQL] support writing bucketed table","user":{"login":"cloud-fan","id":3182036,"avatar_url":"https://avatars.githubusercontent.com/u/3182036?v=3","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":28,"created_at":"2015-12-28T14:28:58Z","updated_at":"2017-01-18T19:27:24Z","closed_at":"2016-01-07T00:59:36Z","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/10498","html_url":"https://github.com/apache/spark/pull/10498","diff_url":"https://github.com/apache/spark/pull/10498.diff","patch_url":"https://github.com/apache/spark/pull/10498.patch"},"body":"This PR adds bucket write support to Spark SQL. User can specify bucketing columns, numBuckets and sorting columns with or without partition columns. For example:\n\n```\ndf.write.partitionBy(\"year\").bucketBy(8, \"country\").sortBy(\"amount\").saveAsTable(\"sales\")\n```\n\nWhen bucketing is used, we will calculate bucket id for each record, and group the records by bucket id. For each group, we will create a file with bucket id in its name, and write data into it. For each bucket file, if sorting columns are specified, the data will be sorted before write.\n\nNote that there may be multiply files for one bucket, as the data is distributed.\n\nCurrently we store the bucket metadata at hive metastore in a non-hive-compatible way. We use different bucketing hash function compared to hive, so we can't be compatible anyway.\n\nLimitations:\n- Can't write bucketed data without hive metastore.\n- Can't insert bucketed data into existing hive tables. \n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/273575049","html_url":"https://github.com/apache/spark/pull/10498#issuecomment-273575049","issue_url":"https://api.github.com/repos/apache/spark/issues/10498","id":273575049,"user":{"login":"tejasapatil","id":3230057,"avatar_url":"https://avatars.githubusercontent.com/u/3230057?v=3","gravatar_id":"","url":"https://api.github.com/users/tejasapatil","html_url":"https://github.com/tejasapatil","followers_url":"https://api.github.com/users/tejasapatil/followers","following_url":"https://api.github.com/users/tejasapatil/following{/other_user}","gists_url":"https://api.github.com/users/tejasapatil/gists{/gist_id}","starred_url":"https://api.github.com/users/tejasapatil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tejasapatil/subscriptions","organizations_url":"https://api.github.com/users/tejasapatil/orgs","repos_url":"https://api.github.com/users/tejasapatil/repos","events_url":"https://api.github.com/users/tejasapatil/events{/privacy}","received_events_url":"https://api.github.com/users/tejasapatil/received_events","type":"User","site_admin":false},"created_at":"2017-01-18T19:27:24Z","updated_at":"2017-01-18T19:27:24Z","body":"@infinitymittal : It will take time to have a fully functional support added. I had initiated a design proposal to get consensus on this could be done : https://issues.apache.org/jira/browse/SPARK-19256\r\n\r\nIn Spark, \"hive.enforce.bucketing\" is not respected. #15300 won't guarantee that the data written adheres to Hive's bucketing spec so approach taken there is to fail in user sets configs to enforce bucketing. This will avoid wrong data being written when user is expecting correct outputs after setting \"hive.enforce.bucketing\" to true. The longer term plan is to get rid of these configs and always write properly bucketed data (hive 2.x follows this model)."}}
{"actor":{"display_login":"tejasapatil"}}